I had the same problem about 8 years ago.......2 years after I quit going. Just don't answer the phone unless you know who is calling (caller ID) and don't answer the door (peep hole).
However......the PO caught me once as l came out from the rear of the house from doing yard work and he just pulled up. I was like a deer in the headlights then. My excuse was that I was still recovering from getting broadsided on my Harley and told him everything he wanted to hear. That was 2003.
These days.....if anyone sees me since I've moved.........I'm totally free and will lay it all out for them. No more fear!
HappyDad