If you were unlucky enough to be elder, MS or even a mic handler, then there was usually some chore lined up for you to do when you showed up at the hall. If you weren't going to be there, then you needed someone to cover.
No big deal for mic handlers or attendants, but MSs had other chores, liking collecting the collection box money, distributing literature or mags, etc. So, we had to call around to get someone to cover.
Even still, we didn't need to check in with an elder. And if the chore was handled in you absence you never heard anything. It was when you didn't show up and no one covered you recieved a phone call the next day or so.
Rock, you say the elder sees potential in you, which leads me to believe that he wants to groom you to be an MS. If you're not being used in the hall and this elder wants to hear from you when you're not coming is way over the top. Then showing up is even more over the top.
Going by your description of events, this elder is going to be trouble. You have to stand your ground now. Every inch you give, he'll just take more and more libertys. There's a point where you have to stand your ground and refuse to let him bully you.
The response of "I didn't want to" was actually a good response. You don't need to justify your actions. It was direct and to the point. He can't read anything into that response. Walking away was good too. It signaled, on no uncertain terms, that the conversation was over.
Sometimes the simple and direct action is the best.