I think being away from the cult and spending more time around normal people is having an impact on me - it suddenly strikes me as odd that someone that you're not very close with would ask to come to your home for something that is merely a social visit. It seems to me that if they want a social visit they should invite you to their home - they're the ones that want it, so they should be the ones to have to go to the trouble of preparing the home for guests, handling entertainment, etc. It's not a purely social visit - if it were then mr elder would be fine with hosting - it's because he's doing this as a part of his duties that he's looking for a way to reduce the load on himself by inviting himself to your house. The other advantage to the elder in this case is that by meeting at your home he's able to have you cornered - it's much harder to come up with a polite excuse (and most people will actually go a long way in the effort of following social norms of politeness even when the other party is not) to terminate the visit when they're in your home. You can't make a vague claim of other plans or give a reason you need to excuse yourself and leave, you have to kick them out of your home. So, as counterintuitive as it might seem, having the visit at your home actually puts a fair amount of the power in their hands because it forces you to be an impolite and ungracious host if you want to control the length of the visit.
It might be interesting to say something like "well it's actually going to be very difficult for us to have guests at our home right now, even for just a short social visit, but we might be free to drop by if you'd like to have us over. It might be interesting to see how they'd react to that, and if you do end up doing the visit, it will be much easier to politely excuse yourself if things get uncomfortable.