I see that Joseph asked a question a while back that I did not see before.
It seems to me that me’ikar hadin the consequences to the person don’t matter; what matters is that your intention is not to harm him but only to stop him harming you. That doing so will result in harm to him is his problem; he should have thought of that before he decided to harm you. Of course, in practice ahavas yisroel, ve’osiso hayoshor vehatov, and also Pareto optimality and simple mentchlichkeit (all four of these may amount the same thing) mean that you should balance the harms involved, and not subject someone else to a great harm merely in order to prevent a small harm to yourself; instead you should suffer the small harm and then try to collect it from the offender. If necessary by telling him that if he doesn’t pay then next time you will call the police. But if you do decide to just call the police I don’t think that makes you a mosser.
This also goes to answer your second question, about what I mean by a reasonable effort. Again, I threw that in not because I saw it in the Shulchon Oruch but because sevoro hi. It’s not right to subject someone to harm without first trying to resolve the situation. But you can’t be expected to take heroic measures just to save another person from harm. If a reasonable effort won’t do it then you’re entitled to save yourself at his expense, since he’s the aggressor. So what is a reasonable effort depends on the circumstances and your available resources, which include time and patience. It has to be reasonable to you.