This seems to be a popular topic. Our experience never allowed for a choice. We walked away, even moved...They pursued us and collected odd testimonials from people all over the country. Maybe we should have been flattered that we merited such attention. lol. I'm sure who we were made a great deal of difference.
Sure, if you can walk away, do it. I personally never felt the need to tell anyone off or write a manifesto detailing my views to people who didn't want to hear.
In the end we didn't have to lie to people, nor did we say stuff we would later regret. Control what you can but be true to yourself. Don't become someone you dislike to be liked by others.