First I think you should seek professional (Rav and or therapist) help for this with or without your son and spouse. As soliek said it sounds as if you are stressed and deserve to help yourself as well. Suggestions here are good but you need to have a tachlis conversation with someone who is experienced with helping parents and families deal with similar issues.
That said I want to try and contribute my own suggestions for your consideration. I agree with what everyone here said about unconditional love. Does your love have boundaries? Can you conceive of situations that would cause you to lose that love? If yes then I would certainly bring this up with a (frum) therapist to discuss privately. If not then your challenge is to make your son understand this. He should feel that no matter what he can always come and speak to you and you will do your best to help in the situation. This is not a simple thing but it is important.
Second is honesty. I dont agree with posters saying you should feign interest in something that doesnt interest you (e.g. rock music). Children are very good at detecting when they are being played. On the other hand I think he will increasingly respect you more for sticking to your principles. So when I said above that you would do your best to help, it does not include breaking your own guidelines – it does mean that you will listen, try not to judge, and help within halchic parameters. But no matter what, you are open to hear and will always love.
I think your discussion with your child should be along the lines of:
1. Will always love you
2. Want you to speak to me (us) about <<anything>> that is troubling you and we will always listen and try to help
3. I will never compromise my religious principles and will always be honest with you if you are doing something that i dont approve of (if asked or if I think it will help) but NEVER in a heavy handed way.
4. I have certain guidelines that you must adhere to if you are to remain in my house (and if you do I will always have my door open to you). If you choose not to adhere to these guidelines and to leave I will always be interested in where you go and what you are doing and want to stay in touch as much as possible. If leaving is the resolution it should not be done by running away but by mutual discussion as much as possible.
To your other children I would say that punishment is a tool used to keep children following parental rules. unfortunately their brother is beyond the help of punishment and that is very painful to you. Nevertheless you will always love all your children and very much want your child to return to Torah and as long as certain rules are followed this house will have to accept his presence (and show same love and kindness shown to rest of family). This does not mean in any way that this behavior is approved of.
These are very difficult and delicate discussions and I think should be had (if at all) under advisement of an outside expert.
Hope this helps and good luck!