The imperfect men card drives me nuts. It is the bandaid to cover all sins along with the Jehovah will sort it out in his own time.
When my mum uses that I just say to her what makes them any different to any other imprerfect men who think they are doing Gods work. There is nothing that sets them appart at all since although you say they are appointed by holy spirit they still make the same mistakes as other religious leaders and abuse power.
Since you leave it to god who is supposed to judge all men who claim to be leaders of faith then they are no different to anyone else.
The holy spirit is so often asleep at the wheel as to not be there at all.