Lots of sound advice here, flag. It appears you are listening to it. Your original post gave the impression the elder asked to meet with you alone. Not sure if that's true, but if so it's a big no-no in the oganization. An elder should never ask a sister to meet him alone for any reason. You could simply say it's inappropriate to meet with him without your husband present and ask if he'd like you to ask your husband to join you. If he says yes, tell him you will talk to your husband about it and get back to him if your husband feels up to it.
I was an elder for more than 20 years until recently when I resigned and walked.
I completely agree with the poster who said this is a boundary issue and that elders inappropriately assume they can stick their nose in your business whenver they want to. If you sidestep this, they won't like it much but they will eventually get bored and move on. But only if you do not raise any "apostate" flags.
When I explained to an elder that I wasn't getting to meetings (and it had been six months), he probed for the reasons. I simply told him about some serious health problems we'd experienced in our family and laid it on that.
I was pretty convincing, I guess, because he sighed a big sigh of relief and said, "Oh, so you're all right then, great!"
I wasn't "all right" physically, as I had just explained in detail. But in his mind, I was "all right" spiritually in that I had not expressed any doubts or lack of faith. He never came back. That's been 18 months or so.
That's how their minds work. They pigeon-hole the "problem" JWs into "trouble" and "no trouble." If you can get yourself placed in the latter category, you're home free.