It is a huge aveira. First of all, if it is an aveil’s kaddish, it is disrespectful to both the mourner AND to Hashem. Second, someone is affirming Hashem’s greatness, and some boor has to TALK at that moment??? I knwo that whatever I am doing when I walk into my Shul, if I hear Kaddish, I stop in my tracks and answer amein. I go to a shiur once a week at night right after the men daven maariv, so I typically hear Kaddish being said more than once. Though I am not in the minyan obviously, I stand up, stop my conversation and respond appropriately to the Kaddish.
My Rov told me there is written somewhere in E”Y on a stone or something, that one who talks during Kaddish loses all bracha (meaning gets a k’lala chas v’sholom). Has anyone ever heard of such a thing?