I never felt contempt for non-Witnesses when I was a Witness. Probably because, much to my parents' dismay, I always maintained friendships with non-Witnesses. In fact, throughout the 30 some odd years I was in many of my best friends were non-Witnesses.
I recall clearly being counselled (I think it was a Watchtower study) that we were to refrain from calling non-Witnesses names like "worldly people." The Biblical reference comes from the Pharisees who used the term "am ha·ʼaʹrets" or "people of the land" as a derogatory slur. The Pharisees used this term to refer to those who did not observe Jewish customs. Sound familiar?