You are smart to question the situation you are in.
Do not worry about what people will think, or that they will equate this situation with shidduchim.
Most people won’t know you had a job that you left, do not put it on your resume. It will soon be forgotten. With regard to people who do know you are working, either explain the situation and they should respect you for your decision. If they are the type you are worried about, just say it wasn’t a good place for you/ you didn’t like the job/ your boss hired his niece…… It will soon be behind you. Hashem will repay you for acting l’shem Shomayim – only good will come out of it.