Maybe it's what your dad sees as a subtle way to get you to
commit yourself either way. If you still have any feeling for the
watchtower, you will go back, if not you will reveal this to him,
and the boe will then be able to take what they consider the
appropriate action, ie df'ing you.
I very much doubt it's an off-the-cuff remark by him, there is
likely to be a reason for it, and I'd be willing to bet that your
name has come up at an elders meeting already, which could
be why the elder you saw the other day was uncomfortable at
being around you while his wife was friendly.