Because of the organizational structure and the power and responsibility given to elders, you create an environment ripe for abuse. The structure also makes it so that the congregation members are very likely to side with (and make excuses for) the elders if they cross any lines. They were, after all, put there by God's own hand.
And that abuse does not need to be carried out with intent. Sincerity is often more dangerous because the sincere person has no reason to back off or try to make an accommodation. I knew a number of men who were normally very humble and very pleasant people, and they saw the position as something sacred and important, as a way of serving the congregation. Yet that attitude made them some of the most relentless and hard-line people when there was an issue to investigate. They did not feel as if they were crossing lines, they felt they were doing everything necessary to protect and preserve the congregation.
"For the greater good" has been used to excuse bad behavior for all of human history. There is nothing more dangerous than someone who will burn down everything because they feel that it is the right thing to do.