I know some women are XXY and some scientific sources that some of them are fertile - this is all very well but these people are statistical outliers. They are well outside the average.
And in science, averages are very important. If an alien from outer space came to Earth and asked us about men and women, we'd say that women have XX chromosomes and get pregnant, because that's how the average woman is. We'd also say that men have a penis and testicles and produce viable sperm because that's what the average man is like.
I think you're doing two things: you're downplaying the importance of sex chromosomes, plus you're being a bit dishonest with statistics in that you are attaching too much importance to statistical outliers.
'The phenotype comes from the genotype' - my former plant science lecturer, Dr Ianis Matsoukas.