I don't know your exact situation, but I've found that Memorials are so stressful for elders, with the preparations and everything else going on, they really don't have time to harass anyone. Other than some love bombing, I wouldn't worry about too much interaction with the elders, especially anything like being pulled into the back room for a talking to.
And if you felt comfortable that you wouldn't be bothered like that, then maybe you could go for your mom's sake. It's one hour, once a year.
I used to have anxiety attacks about going, but not anymore. And yes, I'm going. It's a compromise. One one hour meeting a year isn't going to kill me. And there's nothing they can say or do in that one hour that's going to make me go, 'oh my god.. I was wrong, I should come back before it's too late'. No, I am now immune to their cult tactics and BS doctrine. But, my situation isn't the same as yours or that guy's or that gal's, so what I do may not work for everyone. You have to weigh all the circumstances, and make a decision that you can live with.