A 1/5th means up to 20% of one’s total income.
Given that a quarter is way more than a “Shoveh Perutoh”, one certainly does not have to any Schnorrer more than a quarter.
One can choose to whom s/he wishes to give his/her nice big donations from his/her Ma’aser money, which may be a Yeshiva one feels connected with, or an organization one feels close to.
If this bothers the schnorrer enough to throw the 1/4 back in your face, that is their Chutzpoh Atzumoh. All the 1/4’s eventually commence to add up.