Things may well depend upon the status of the congregation that you visited.
In my case there were a lot that coincidently drifted/left/disfellowshipped/dissasociated/faded, and of those that were left behind, a substantial amount of them were either born into it through generations and didn't have the guts/means or inclination to leave and there were also others that simply half-heartedly went with the flow, (i.e. needed constant reminders via special needs talks to make them answer up, go out on field service or simply attend meetings)! I'm sure my congregation wasn't specifically different to any other!
Therefore, in my case, when the elders came-a -knocking, I simply told them how it was for me, I didn't hold back or give them any bullshit and as a consequence, they were unable to pursuade me to return.
I can understand your fear, I have felt that before, yet with time I am sure that you will be able to focus upon what is right for you and not what is right for them, also, the advice that has been previously given here with regards to 'being out' or 'having something else to do' at the time of the arranged appointment with them is something to be worthy of consideration?
Whatever suits you is what you need to go with.
Take care,
Regards,
Bay64me