Well maybe there are cycles of ups and downs. In the pursuit of money, you lose your culture by importing so many workers.
You only need 1 man to impregnate hundreds of women. It fixes itself out when all the men die off. Important worthless migrants to fill the void during the low cycle is the nefarious part.
Importing*
The top 10% of men got all the women through history anyways.
So what should've been my reward for making it past 40 was a bunch of women for my Palace, is now replaced by Gonzalez, Tyrone, and Herpjeet taking my flock.