I think if you disassociate yourself you are still playing by their rules and their terms.
I think some here have avoided it by saying things like they've been "dealing with it on their own for a while and will just continue to do that," or "I'll talk when I feel more comfortable".... or you could say you would talk when you know what you want to talk about!
You could mention you are under lots of pressure now and will get back to them when you feel up to it.
You could move. Although I wouldn't. You could change your phone number and then only give it out to ones you want and get caller ID.