When something is assur mita’am minhag, often the minhag was never to be machmir b’shaas had’chak. An example that comes to mind is waiting six hours for Ashkenazim, where we are more meikil for cholim than Sefardim would be.
When it’s assur, though, it’s assur.
If the Rema and poskim say it’s assur, you shouldn’t be afraid to say it’s assur. The fact that there’s some more “wiggle room” doesn’t change the fact that it’s assur.