I read an article last year (and for the life of me I cannot remember where) that no matter how much of the service you know by heart and therefore do not use the siddur, you should always use it for every Kaddish in case you forget a word.
Personally, I would say that I do some parts by heart and use the siddur for some parts. I wish that I knew more by heart because I enjoy davening with my eyes closed. I seem to feel closer to Hashem that way. (now I wish I could learn to keep my eyes open during the sermon).