Glad to hear of these great experiences.
The WT poisons the minds of JWs against churches -- satanism, demons, Babylon the Great, snake handling, greed, hypocrisy, etc.
It's also easy to project the legalism of the WT onto the churches and assume that all of them are the same or worse.
Since you found that pastor to be loving and helpful and not pushy, I'd suggest cultivating a friendship with him.
He seems to understand your need to process things in your own way and at your own pace.
If you ever do attend a service at his church and get upset by something, talk to the guy about it so he knows what's bothering you.
Remember that a lot of Christians are totally unaware of things that would be negative triggers for someone with a JW background -- crosses, collection plates, praying to Jesus, calling Jesus God, emotional services, and the like.
All that stuff is just normal to them.
So if you encounter stuff like that, don't just run away; talk to this pastor about it; sounds like he's an understanding and caring guy who needs your help to understand what you're going through and working through.
If you'd like to PM me about any of what I've said, feel free.