After further review and thought, I really don't see how removing a monument or statue that may be truly offensive to a large group of people is in some way rewriting history.
I think of Germany or other parts of Europe. You don't see swastikas on buildings or statues of Hitler or Mussolini or other upper level generals in public. Would it be good for a person (think Jewish) to drive around a city and see Hitler in a public park, even though it is "historical?"
As an example, if you are in Munich (Munchen) Germany and drive 20 minutes or so out of the city you can go to Dachau (concentration camp) and tour the place for a few hours. You will see all of the Nazi stuff, the barracks, swastikas, gas chamber, ovens, pictures, you name it.
From an historical perspective it is definitely worth the visit, though very creepy, at least to me. But I can understand that the stuff is not for everyday viewing or some sort of feeling proud about what happened. Wouldn't it be offensive to many to see a statue of Nazi Dr. Mengele in public?