What if ((their)) obsession with children began with the fallen angels.
We're told they found the daughters of men attractive but we are not told their ages. Periods and therefore fertility can began as young as 7 y/o.
The giants came from angels taking an earthly form and coming into the daughters of men so maybe the Nephilim bloodlines carry the same perversions.