I've always found that this situation causes a sort of panic for Witnesses, when you clearly are not living the JW lifestyle, but haven't been officially black-listed.
I visited my Elder Father and wife about 10 years ago and were invited to the meeting with my non-JW ex-husband. (I think my Dad was hoping to make an impression on my then husband and lure him in) I had some old friends that still attended the meetings there, and wanted to see them so agreed to go. I've never seen so many darting eyeballs and foot shifting in my entire life. People didn't know whether they should hug me for appearances sake or run screaming the other way for the worldly bad person I was. It was actually quite funny.
I think they're just worried our Worldly Cooties may rub off on them.