DaasYochid, no, I could not come up with a better analogy. Regardless of how commendable some of the people who serve in the IDF are, they are part of an organization that is anti-Torah which is, in turn, a part of a State that is anti-Torah. So I thought my analogy was actually pretty charitable.
Regardless, as you mentioned at the end of your post, it doesn’t matter what you might think your reason is for wearing a sweatshirt or anything else.
If the garment carries with it a popular connotation then by wearing that garment you will convey that connotation even if you’re doing so because it’s “stylish” or comfortable or you like the color or for any other reason.
Wearing an IDF sweatshirt necessarily implies endorsement of the IDF and, also by extension, the State of Israel. This is, therefore, inappropriate, on many levels, as mentioned earlier.