In Scripture, men are also called "sons of God".
That is not talking about "angels" mating with men.
Nephilim refers to "the accursed ones".
The name alone says it.
What line of men were accursed after Adam?
Cain was accursed.
On top of inheriting the sin nature from his father, Adam, his was a specific curse additional curse.
Cain had been separated from all the others, a wanderer, and started his own nation.
After a time, they began to steal the daughters of the others due to their curse.
It is the accursed ones who called themselves "sons of God", but not sons of the true God which Cain had long rejected.
Cain and his heirs deified themselves, and they became the model upon which all the gods of the heathen were fashioned.