Assume that every time I go outside, there is a 95% chance that I will make a positive impression on the public by doing such things as feeding the sick, healing the hungry, and helping old women across the street. There is also a 5% chance that I will inadvertently bump into someone, fail to notice the middle aged man who doesn’t have a seat on the bus, or drop something out of my pocket. The expected gain from someone judging all Jews positively based on observing my positive behavior would still be lower than the expected loss of people judging all Jews negatively based on observing my negative behavior.
Therefore, it makes sense to wear a cap.
However, an argument can be made that in those situations you look more Jewish with a cap than with a yarmulka, because obvious Jews with Yarmulkas can be anything from MO to Eida Hachareidis, but obvious Jews with caps all look chassidish.