I can't believe how understanding you are. It's one thing to decide that you are suddenly unsavoury characters and that they need the elders ok before they eat with you - quite another to decide this ten minutes into the dinner engagement. Rude is not the word.
That's how any normal person would look at the situation. But from a JW point of view, if you are "bad association" then you must know it yourself, because it's like a real fact about you, rather than something the elders simply made up. And if it's a real fact about you that you are "bad association", and you attempted to associate with them despite your bad status. Well in that case the fault is all yours. In fact you owe them an apology for attempting to associate with them and putting them in this terribly awkward situation!
That's if you have a proper theocratic appreciation of the situation.