I agree that we are only judged according to what we are capable of, and getting angry when your kids make you mad, is really not the worst thing I’ve heard of.
At the same time, I would think you need to ask mechila, even if G-d really isn’t going to count it against you.
For example, imagine you tell lashon hara about someone and ruin their marriage. Imagine it was in a way that it was assur, but it was so hard that there really can’t be a complaint against you (like maybe you were so mad about something and it was in a fit of rage that wasn’t your fault). I still think you better apologize.
So again. It is hard to not get mad at your kids, and it is probably not very high on your aveira list. But if you hurt someone, you should apologize.