The rapidity of it all after the Flood really is the problem, Phizzy. Life's distribution after the Flood. Mankind's growth after the Flood, so that already 200 years later nations and cities are mentioned.
If saying the Flood was devastatingly huge, enormous, never seen anything like it, massive results all over the world - but not covering all of the Earth, the problem would be solved.
Millions of years prior to Adam doesn't matter, as long as it all narrowed down to a handfull of people and animals 4,200 years ago.