The weirdness all starts from trying to apply modern principles of reason and science to this ancient text. The ancients knew it was allegory. It's only the foolishness of modern man that tries to make sense of it.
It's a "modern" thing really.
WHile some of the ancients might have viewed some parts as "literal fact", since there was no reason to NOT view it as such at the time, many, like Augustine, kept a very opne mind to the "literal and concrete" reading of Genesis.
Of course when we say "read it literally" we mean "literal and concrete" as opposed to what they meant and what we SOULD mean, which is to read it in its literal genre.