I have never had trouble making friends. Also, I have never been a witness. But, I think I can make a contribution.
I have this friend whose sister married a nice man. It was kind of odd to us, however, that, when he married, there was no one from his side of the family. Only years later did I learn that he had been a witness who left. He happened to be marrying a Baptist. He was just like everyone else, and, what's more, he has the reputation of not being meddlesome and not being a gossip. Not that he doesn't have any personality flaws, but for the most part he is easygoing and kind.
I think you just have to be yourself. People won't care at all whether you were a witness or not. In fact, if you look at it carefully, the reverse is true: it's the witnesses who have something against "the worldlies". I find that for many witnesses, leaving that "us vs. them" mentality is a big hurdle. Even here, if you are a worldly, you notice that sometimes the witnesses are still seeing the world as "us" and them. It's not a surprise, since they were raised to understand the world in such a way. And, people of all sorts do tend to do groupthink.
If you are a nice person and have normal interactions, then friends will come your way.
I can understand that meeting former witnesses who really-really understand you is a valuable thing, but in my humble opinion you should not constrain yourself to mingling only with former witnesses, those who you feel to be just like you. I have chosen the final words of the previous sentence very carefully to make my point. If you are to escape the prison, then everyone outside is your friend, in principle. If you're not into heavy metal or poetry you won't make it with those groups, but you'll know that only after you meet them.