Privileges schmivleges. Beyond the touching the boobs, you don't have to say anymore.
Yes, they can take away such a privilege just based on an accusation and your denial to meet and clear it up, or based on your confirmation of the accusation. I think you already came to terms with losing your pioneer status and will be fine with that.
If you desire to avoid a judicial committee, you can deny anything else happened. No matter what the guy says, you say "HE WISHES THAT HAPPENED." Stick to your guns with "I never saw him again and went straight to Jehovah with this."
You may never get past the judgement of the elders and the whispering. So make what you will of it and get on with your life. I hope the best for you.