I know of two brothers who were elders and are no longer because they recognise faults in the doctrine and practices. They still are (AFAIK) victims of cognitive dissonance however and remain essentially "in" because they square the circle mentally with reasoning like "go back to the fundamentals - do I believe there is a God and he has a purpose. Yes. Then the rest doesn't matter".
I think there is a growing number of brothers like this and, yes, some of them will remain as elders and servants for some time.