An ’06 essay says falling fertility often afflicted indulged elites:

“If we could survive without a wife, citizens of Rome, all of us would do without that nuisance.” So proclaimed the Roman general, statesman, and censor Quintus Caecilius Metellus Macedonicus, in 131 B.C. Still, he went on to plead, falling birthrates required that Roman men fulfill their duty to reproduce, no matter how irritating Roman women might have become. …

Throughout the broad sweep of human history, there are many examples of people, or classes of people, who chose to avoid the costs of parenthood. Indeed, falling fertility is a recurring tendency of human civilization. …

Societies with high fertility grew in strength and number and began menacing those with lower fertility. … That was the lesson King Pyrrhus learned in the third century B.C., when he marched his Greek armies into the Italian peninsula and tried to take on the Romans. …

Greece, after falling into a long era of population decline, eventually became a looted colony of Rome. Like today’s modern, well-fed nations, both ancient Greece and Rome eventually found that their elites had lost interest in the often dreary chores of family life. “In our time all Greece was visited by a dearth of children and a general decay of population,” lamented the Greek historian Polybius around 140 B.C., just as Greece was giving in to Roman domination. “This evil grew upon us rapidly, and without attracting attention, by our men becoming perverted to a passion for show and money and the pleasures of an idle life.” (more; HT sestamibi)

This supports the idea that farmers naturally return to foraging ways as they get rich. I wonder how many “civilization collapses” were due to fertility losing its social status.