I agree with several of the others... if you're not interested in going back or being harassed, do not talk to them.
Why is it that when anyone does visit someone from the khall it's always in their dress clothes? Because they are out in service and can still keep the meter ticking when dropping in on someone. If they are dressed in street clothes, why that would be a waste of time, uncountable time.
I remember a friend dropping by one Saturday morning. I had known this guy for probably 20 years. He married a child hood friend. We had had some good times together over the years though I had kind of lost touch with him as I faded into obscurity. He said he was in the neighborhood and just thought he'd stop by and pay a visit. We talked about certain people and our jobs and our parents...all the small talk you have with someone you bump into at the mall.
Typical scenerio of someone taking the time to look up an old friend, right? Nope. This old friend was an elder in my last congregation. He was working a couple of streets over and decided he would use the oppurtunity to come by and 'encourage' me. He was dressed in his suit. He was carrying his bookbag. This was not two old friends catching up. This was an elder on a recon mission. He feigned glad tidings in order to get inside and feel me out. For those of you uninitiated in this experience, make sure to keep them on the porch or in the entry of the house. Don't invite them in, sit them down and offer coffee. No no no... And don't answer questions. I was able to keep this guy off balance by talking about any and everything not related to the KH. We knew so many of the same people by the time he left, I was up to date on all the whereabouts of old friends but he never got a question in about my situation and if I wanted/needed help.
After that I thought about just showing up at his house one evening right after work. Inconvenient for him, since he would be getting ready for a meeting or dinner...but hey, I'm in the neighborhood, why not? If he thought so much of me to stop by my house, why, I should return the favor one day and visit him. Good times we'd have, I'm sure...