If I remember correctly, the first christians in Rome used the drawings of a fishlike symbol. This symbol has been uset in the ancient rom catacombs.
Historicians belived that It was a secret sign, to gather all those who secretly belived in Jesus. Remember the Bible story of the "fisherman"..!
Yeah its called the ICHTHYS
A Christian would draw one half of the fish symbol in the dirt and the onlooker would complete the symbol to indicate that they were both Christians.
"Ίησοῦς Χριστός, Θεοῦ Υἱός, Σωτήρ", (Iēsous Christos, Theou Yios, Sōtēr), which translates into English as "Jesus Christ, Son of God, Saviour"
However using the OP's logic one could argue that the fish was used for centuries before that in pagan worship, so one would have to conclude that it was also pagan. Yes pagan symbolism can be kicked out behind every rock but its sort of ridiculous because symbols mean whatever meaning we ascribe to them.
Also paganism existed way before Christianity and so all symbols used since then can be construed as pagan.