The idea that “We have never added memorial days for tragedies that affected only a portion of Klal Yisrael” Is nonsense. 20 sivan being but one example (though it is not widely commemorated today, some siddurim still have selichos (!) for 20 Sivan printed in them.
The real reason the charedi world doesnt commemorate yom hashoah is because it was instituted from the get go as a bridge to yom haatzmaut it is designed to serve as a contrast between the galut vs geulah. This obviously did not and does not sit well with the Charedi velt.
The second reason is that the day is not just a day to serve as a memorial to the victims. It also serves to glorify the armed resistors. The full name of the day is “Yom HaZikaron laShoah ve-laG’vurah” The chardei world by and large did not participate in armed resistance and thus felt somwhat excluded and even slighted by the notion.