Your story is a lot like mine. I was "raised in the truth" more or less, since my parents got into it when I was 11, and quickly cut off all other association for my brother and I, even extended family was very much distanced at the time.
I've reconnected with my "worldly" family since fading years back, but strangely, my Witness parents have also reconnected with worldly siblings now that they're older after all those years of avoidance. Hm...maybe it's an age thing, they're all in their 70s now and thinking a lot about the Grim Reaper coming knocking!
My parents also know I haven't been active in ages. They also know I have "issues" with WTS teachings and have for some time, but they don't avoid me, although I do live far from them. But, they call every so often. We never discuss "the Truth" nor do they try to Witness to me anymore. We talk about every day normal family things. We sort of have a "don't ask, don't tell" thing going on...I don't tell them about my non Witness-like activities and they never ask or come close to asking.
I think the biggest problem in finding non Witness friends for people who are DF'd, fading or just inactive is that it's really drilled into you to FEAR "worldly" people.
I was afraid they were all minions of Satan just waiting to do evil things to me, or get me into evil things, and gee, what a shock to discover many "worldly" people I've met along the way are nicer, more spiritual, more Christlike (even if some of them aren't churchgoers or even religious) and way more interesting that many Witnesses I know or knew. Go figure!
Of course, there are plenty of jerks and morons and downright nasty wastes of skin in the world, but the Witnesses have their fair share of those too, in case you haven't noticed. Or maybe it was just me that ran into most of them. LOL