Hi auntiem, I think it was inappropriate for the elder to show up alone at your door. Was he there on a shepherding call? Dressed the way you describe it, I doubt it. Is he a close friend of the family? Still it would be inappropiate to show up alone knowing that your husband is not at home, but it could explain his thinking.
I think that the visits that the elders are scheduling between January and March are not to get you to disassociate, but to help inactive Witnesses to become publishers again. They might encourage those inactives ones to accept a bible study once again to strengthen their faith. No one should be scared of the elders. If they are taking the time to visit inactive ones, then that is proof that they are concerned and remember those that have left the flock. Of course, it's still your right to accept or deny a visit. But try to avoid a confrontation. I have found that sitting down to talk over a hot cocoa or coffee goes along way in clearing up differences.
I hope you are there to receive the elder once he comes around again, of course only if he's accompanied by another elder, if alone it's better not to open the door. Try having a calm, rational conversation with them without attacking or being on the defensive. Looking up scriptures and talking spiritual matters never hurt anyone. I hope you have a good day.