Same here Pale emperor. I've been fading for a few years now and only just recently became inactive. Hubby's an elder. However, I don't feel I owe anybody an explanation as to why I don't do certain things. If I'm asked for my report I just say I haven't got one. If I'm asked why I don't go to the meetings I just say I don't want to. It's a free country, I'm a free human being with free choices that God gave me. No one can make do anything I no longer want to do. I haven't lost my family, possibly because I've taken it slowly and they can see where I'm coming from, even if they don't agree with me. All it takes is to be quietly assertive.
Also, whatever you do, don't accept a shepherding visit. Say you'd rather not talk to anyone. No explanations, just say no thanks.