I can't speak for your local area congregation, but chances are that the elders don't want to bother you. Here, us elders are so busy that we don't have the time or interest to bother someone like you. Only when the CO comes is there much of an effort to try and 'fix' drifters.
As far as the new neighbors, maybe they are spies. Or perhaps they are so desperately lonely that they're trying to find anyone that's not completely destructionworthy to befriend... and count their time talking to you.
Anyway, if you really are concerned, as already mentioned in other posts, you can use the new, "I'm concerned about news reports of pedophiles in the JWs."
And 'no' you're not a coward. You are the one choosing your course, and I'm not your judge.