I definitely would look at it as small flag but wouldnt drop it like that! Maybe he was uncomfortable to say about quitting on a first or second date not sure as to how you would take it and his coping mechanism was to talk is if everything was regular.
What is important in relationships is communication so if you have no other reason to say no, I would say go out again, AND be bold and bring it up… you need to tell him that you know he isnt working and that you need him to be open with you and honest with you. you can see how he reacts to that and see other important parts of his personality. I can understand why he would have felt uncomfortable – not that he was right but its not necessarily a red flag – tho look out for more red flags.