Gedolei Yisroel, including Rav Eliashev, have explicitly and repeatedly cited antagonization resulting in anti-Jewish violence as one of the reasons it is forbidden to go up to the HH”B.
That is correct. This is a second halachic reason, and this is not something we’ve made up.