So she calls you, *twice*, clearly with the intention of scolding you and yelling at you, over a promise you made to a complete stranger when you were in high school? And then she tells your brother that you upset her???
If I could diagnose that $h!t, I'd say your mother has bipolar disorder.
The thing is, if you do as your brother asks and cease communication, they control the narrative of the relationship. "Silentbuddha doesn't call us, doesn't speak to us anymore." All this behind your back and turns you into the bad guy.
If only for your own peace of mind, maybe write a letter to your mom/dad/brother about why the "promise" you made under duress (in a room with other people who had expectations of you to answer the way you did) as a high schooler is not valid. You weren't old enough to drive, vote, get married, get a job, buy or drink alcohol, *enter a contract* (promise), etc. and the law exempts you from those *adult* things because you are still considered to be a child, without the benefit of full brain development. There's science behind that.
You were in an unfair situation. The adults in the room took advantage of you. That was not okay. It's not okay now for your mother to give you $h!t over it. She's likely more upset because she believes that Fred Franz thinks less of her and her parenting.