Also, there is the phenomenon of middle class flight from the city. This is similar to "white flight" in the mid 20th century that left the inner cities black while a lot of whites moved to the suburbs and forever snarled traffic.
However, Baltimore has always been a segregated city. There were unofficial black neighborhoods and white neighborhoods. As fortunes changed for working class whites, especially after this latest recession, poverty increased even among whites. The middle class moved away from neighborhoods that had fallen into poverty, and those neighborhoods were filled in by poverty -stricken folks. Then, those neighborhoods attract drugs and crime and once a neighborhood goes downhill, no one with better fortunes will ever want to move there.
It's like the tragedy of the commons; everyone acts in their own personal best interests which is detrimental to the common interest. And to rebalance that you need outside intervention, whether public or private. Publicly, Baltimore has provided old abandoned rowhomes for a dollar to people who promised to refurbish and live there for a certain period of time. Some neighborhoods turned around. They have also had a razing program for abandoned blocks of rowhomes to rid the city from some of the blight, but they have trouble finding finding for this.
On the private end, the affluent and prestigious Johns Hopkins University and Hospital is still thriving in the city and they have bought up blocks of rowhomes surrounding them to refurbish and house students and medical interns. This has, in general, spruced up these blocks, but the residents are transiet, so not a perfect solution for neighborhood building.