Freaks. I've sort of been in that situation before but not for the swim. Once was an early season of kayaking on the Blue River that comes out of the dam on Lake Dillon in CO. The water temp is usually around 36-39 degrees at that time of year. I got rolled up and had to pull out - dumbass me didn't think a drysuit was going to be necessary that day due to my extreme kayaking skills. I barely made it over to the shore. My skin was turning blue and purple. The other cold water "event" I thought I could breeze on by was the Spring Splash at Winter Park, CO. They set up a big obstacle course on the slope and when you get to the bottom you have to launch off a jump and try to ski aross the large trench they have dug out. The one time I did it, I simply wore shorts, a tshirt, and my ski boots. Quite a few people in front of me never made it across. I knew I probably wasn't going to make it either so I just did a heli off the jump and landed square in the pond. No one helps you get out, so you're trying to walk out with your skis on in chest deep water. To give you an idea of how cold that was - there was plenty of snow floating on top of the water.
I feel for anyone who accidentally falls into near freezing water. I do however laugh at anyone who complains that the water is cold when they jump in on purpose! I could never do a polar bear club swim these days. Twice similar is enough for me.