once again I would say this is too complex a topic to be more than glossed over here.
but here a few comments.
1) even midas harachamim has rules and dinim. It is not random or by whim.
2) midas harachamim refers to during meah v’esrim. L’achar,
there are also some special allowances, but I don’t know if they come under the heading of midas harachamim.
3) midas harachamim is aroused by t’filos, chesed and so on.
even here, there are rules and dimim.
4) In general His judgements are beyond our understanding.
we can never know the whole story that lead to a schar or onesh and we cannot know the full value of either.