What about wearing a button down collared shirt with a colored cloth napkin in the pocket instead of the jacket?
Even people who wear jackets to work may not go to the extent of tucking in a colored cloth napkin.
Make the napkin thing the fancy piece that shows you’re speaking to the King