My family minhag is to wear a mask properly, not to chew in a mask, and keep it so that the huldas from the neighbors will not reach them. So, we have nothing to worry.
If you keep your mask down so that a kezayit of hametz can fall through, or wear it on a beard that holds breakfast leftovers, you need to worry.