My husband was thinking about this the other day. He thinks that the "sons of God" might actually refer to the line of Seth. Adam's genealogy is given in Gen 5, but only as the line of Seth. In Gen. 4 the line of Cain is given, and some of the men were . He pointed out that in Luke's genealogy of Jesus, Adam is called the son of God.
On Cain's side you have Lamech, who had a hot temper. On Seth's side, there was Enoch, who 'walked' with God'. So, perhaps the descendants of Seth (the boys) were attracted to the daughters of Cain.
Interestingly, after Gen 6:1-2, it reads,
The LORD saw how great man's wickedness on the earth had become, and that every inclination of the thoughts of his heart was only evil all the time.
So things went downhill fast!