I think you are right. Though I returned to the meetings and am only managing to go now and again (as thinking of fading again), i personally think they (the boe in each congregation-to a degree) are afraid to rock the boat, particularly if you are known to have doubts and/or you have other members of your family attending; as they are all too aware of the repurcussions. The boe in my hall are probably well aware of my doubts as I spoke to one of them once, but they never bothered me for years during my last long term fade, which I did wonder about.
I beleive that they will only generally go after you if you are quite outspoken, in which case they can then say they have clear grounds to bring things to a closure.
how did you get to hear that elders from other halls wanted to do something about you?? Why were such elders so keenly interested in doing something about someone from another hall??