You had a Pesak, so follow the Daas Toraoh of your local Orthodox Rabbi. Nowhere does the Pesak say, that you cannot leave your hat @Shul, so can still wear your hat during services.
As far as picking up your hat, you didn’t make any mention of walking 4 Amos after picking up your hat before placing hat on thy head, so I don’t see any Hotza’ah taking place.