Just politely tell them to FUCK OFF and mind your own business ,I find it frustrating that grown ups / adults allow themselves to be manipulated ,bullied,and controlled by other people .
Your a man grow some balls and stand up for yourself and your wife.and don`t let some wannabe somebody`s lord it over you.
As has been pointed out numerous times on this board the only power they have over you is the power you give them
Wake up and cut off their power .
If I sound harsh and blunt so be it ,sometimes you just have to be cruel to be kind.
So don`t pussy-foot around with them otherwise you will always be the victim
Surely you and your wife deserve better than that.