I am not sure that it's contempt. I think it's an ingrained fear of getting caught up with a worldly associate. What if it's a neighbour who smiles in a friendly way? They may invite you for coffee, that would be baaaaad.
Better to be aloof and try not to encourage them to befriend us.
The result? An insular weird bunch with a kingdom smile but who seem too self righteous to lower themselves to be genuine friends.