I'm from the south and have lived in rural areas where older blacks and whites could tell you stories about their cotton picking days. When I was a kid in East Texas there were a lot of people who used the phrase but I never got the impression that it was a derogatory phrase. Now Cotton Picker is a different story. Cotton Picker was usually used to refer to someone the speaker considered worthless whether black or white but when used in connection with a black person it was taken as a racial slur. When used to describe a white person it usually meant the "white trash".
In my world, which is probably near the origin of the phrase, Cotton Pickin' was just a colorful addition to a southern story-teller's vocabulary.
How long is a cotton pickin' minute? Just long enough to git a word in edgewise.