On reflection, I am now a bit more optimistic about solving cultural drift, *if* we can get broadly human-level AI, or ems, before our current civilization falls.
if Hanson is right and human cultures have/will become unadaptive, a new adaptive AI culture may be the only way there is complexity/intelligence our piece of the universe going forward
That sounds more like justification for a Butlerian Jihad rollback of AI and modern cultural norms that lead to people not reproducing at population-sustaining rates than a justification for humans letting ourselves go extinct and considering it OK since the AI replaces us.
I think (Lam's summary of) Hanson's beliefs can be used "as justification" for either of the scenario (by which I assume you mean something like "If these beliefs are true, you should put effort towards making this scenario come true")...
... And whether one thinks it's *more* justification for one than the other depends on further beliefs that are unspecified in this particular post, but have been hinted at in other posts (e.g. how likely you believe different strategies might succeed; how under-explored you believe certain ideas are, even if they currently seem lower probability to succeed than well-explored ideas; etc.)
Option One: Superintelligent AI can be invented and successfully Aligned with humanity. Below-replacement birthrates are no longer a problem because the ASI can reverse-engineer biology such that human lifespans are now functionally infinite and any birthrate, however rare, still adds up to positive population growth, by reducing the effort of reproduction with post-scarcity, artificial wombs and robo-nannies, encourage more people who currently wouldn’t consider it worth it to do so, or both.
Option Two: Superintelligent AI can be invented and successfully Aligned with the oligarchs who’re creating it. This solves birthrates, insofar as automation took all the jobs meaning the oligarchs have an endless supply of desperately poor women facing a binary choice between becoming doing anything to become their gold-digging trophy wives or starving. The oligarchy can dispose of economically redundant men on the other hand, by starting a peer power war with conscription like Alex Karp advocates.
Option Three: Superintelligent AI can be invented but Alignment is impossible or at least too unlikely to be worth the risk of trying. Unaligned ASI would obviously get everyone killed. Therefore, don’t build it and treat everyone trying to build it as de facto trying to attack everyone else with a doomsday superweapon, IE, attack them first with overwhelming force before they can succeed and their creation becomes too powerful for such an attack to work.
Option Four: Superintelligent AI is impossible. Civilization continues on its current path and eventually collapse from declining birthrates. Only, our civilization isn’t the only game in town, if we die out all that means is those other civilizations whose different societal norms have maintained their birthrates inherit the planet. Quiverfull, Mormons, Haredi, etc
So much conflation of the word "adaptive"! Usually you conflate adaptiveness of a culture itself (ability of the culture to spread itself through either memetic or reproductive means) with the DNA genetic adaptiveness a culture confers upon its members (ability of a culture to cause its human members to reproduce and survive more.)
But here you're conflating a couple more things: adaptiveness of an AI for capitalism (i.e. ability of the AI to generate profit for its owner, and thereby cause itself to be copied and run on more computers), and adaptiveness of an AI for human culture (which could be either of the previously mentioned conflated senses!)
The elephant in the room is that capitalism is highly maladaptive for the DNA genetic adaptiveness of its human members. Modern capitalist cultures have dramatically lower birth rates. Companies push employees to maximize profit for the company, to the exclusion of social and family life. And they use marketing to cause customers to desire their material products more, which competes with the desire to have and raise children.
I think you're right that there's a distinction between these different senses of adaptiveness that the current post does not make clear, but there seem to be a bit of a similar rhetorical trick with your last paragraph: You're saying that capitalism is bad for one type of adaptiveness (sure, perhaps), and use stereotypically negative-valence imagery (e.g. "Companies push employees to maximize profit for the company, to the exclusion of social and family life") which I think may trick a typical reader into inferring that capitalism is therefore bad for the kind of adaptiveness that we care about.
Maybe it is, maybe it isn't, but I think if you're core point is about how people are conflating different meanings of the term "adaptiveness", it behooves you to put extra effort in being very clear about this.
I don't know your views (or Hason's), but just to give a flavor of how we might move this discussion forward, let's check if we agree on these axioms?
- It would be bad if all intelligence/complexity were to get snuffed out of the universe.
- Making sure that we are "adaptive" (does it matter in what sense? If so, in what sense would we accept vs reject the axiom?) helps reduce the probability of that bad outcome.
If we do agree on these axioms, we can then argue about whether these claims are true (or plausible, if their truth values are currently uncertain):
- Given the current trajectory of most major human culture, it seems like they are not "adaptive" (does it matter in what sense? If so, in what sense would we accept vs reject the claim?)
- "adaptiveness of an AI for capitalism" might be a viable way to preserve "intelligence/complexity" in the universe, even if it is "maladaptive for the DNA genetic adaptiveness of its human members".
> You're saying that capitalism is bad for one type of adaptiveness (sure, perhaps), and use stereotypically negative-valence imagery (e.g. "Companies push employees to maximize profit for the company, to the exclusion of social and family life") which I think may trick a typical reader into inferring that capitalism is therefore bad for the kind of adaptiveness that we care about.
Where's the "trick"? I specified which type of adaptiveness capitalism is bad for (DNA genetic adaptiveness of its human members). There's not much ambiguity in that. Is using negative-valence imagery a "trick"? I think in this case it's just stating the facts.
> It would be bad if all intelligence/complexity were to get snuffed out of the universe.
Agree.
> - Making sure that we are "adaptive" (does it matter in what sense? If so, in what sense would we accept vs reject the axiom?) helps reduce the probability of that bad outcome.
It does matter in what sense. For example, nuclear escalation is "adaptive" when the units of adaptation are nations, because nations that have nuclear weapons can then leverage that into geopolitical power and security against invasion. But nuclear escalation obviously increases the chance of a nuclear war snuffing out human life on Earth.
For another example, fossil fuel usage is economically "adaptive" when the units of adaptation are individuals, corporations, or nations. But it tends towards harmful long-term climate effects for everyone, potentially making human extinction more likely.
Basically there are lots of ways for units of adaptation to be assholes to each other for individual gain and collective harm. That's the tragedy of the commons, or the prisoner's dilemma. If our goal is to keep intelligence alive in the universe as long as possible, we need to set up systems that *align* individual adaptiveness with collective benefit; we should be rewarding units of adaptation (such as humans), when those units tend to improve the long-term outlook of the whole.
The important factor is *alignment*. We need to solve the tragedy of the commons, not glorify and encourage adaptive units doing their best to exacerbate it.
> "adaptiveness of an AI for capitalism" might be a viable way to preserve "intelligence/complexity" in the universe, even if it is "maladaptive for the DNA genetic adaptiveness of its human members".
You mean if the AI results in the extinction of all humans, but the AI outlives us by a long time. To me I think this is a bad outcome. Under, say, preference utilitarianism of current humans, it's clearly a bad thing. I could possibly be persuaded otherwise, but I think you need a very strong and well-grounded theory of ethics to convince me it is a good thing for humans to go extinct in favor of AIs. My status quo bias in favor of living humans is quite strong here. Whatever is "good" about humans, how can you prove it is also "good" about AI, and to what extent? Does one super-powerful AI living a billion years, morally outweigh a billion humans living for a thousand years?
Not "economic freedom" - too vague, too dependent on which government is currently in charge. Rank countries by the actual influence of capitalism, which is GDP per capita.
And yes, you will find a strong negative correlation. The first-world countries with the highest level of GDP per capita also have low birth rates. The poor third-world countries have higher birth rates, though still declining as they become more capitalized and modern.
I'm equating the influence of capitalism with GDP. That's not exactly the same as general prosperity, since I'd interpret general prosperity as the amount of prosperity typical people in the society have. If the top 1% of consumers consume most of the goods and services, then you could have a high GDP without the typical person being prosperous.
Also, GDP does not include services provided outside of capitalism. If your grandmother or stay at home wife provides child care for free, that does not add to GDP, but if you pay a babysitter to provide child care, that does add to GDP.
So GDP is not general prosperity. It's net economic activity. It does correlate with general prosperity, but I would not equate the two.
Remember also to be clear about what kind of adaptiveness we're talking about - in this case the DNA genetic adaptiveness of the citizens.
But it is true that capitalism does tend to increase both general prosperity and GDP in most countries, at least for the time being. And it is also true that capitalism tends to depress the birth rate. This is maladaptive from the standpoint of DNA genetic adaptiveness of the people.
But "maladaptive" is not the same as "bad." What's good or bad depends on what people - not genes - want.
Talks from a few years back featured a "doom disclaimer": civilizational melt-down - unless AI or brain emulations happen first. Then there was a more recent period when this disclaimer was systematically missing from the conversation. It sounds as though we are back to having the disclaimer - and since at this stage it is obvious to everyone that we are having AI first, we are back to being in a sensible position.
>preventing capitalism from evolving more adaptive AI cultures
Why on earth would we want “more adaptive” AI cultures, insofar as the obvious adaptation would be to cut us out of the loop altogether.
if Hanson is right and human cultures have/will become unadaptive, a new adaptive AI culture may be the only way there is complexity/intelligence our piece of the universe going forward
That sounds more like justification for a Butlerian Jihad rollback of AI and modern cultural norms that lead to people not reproducing at population-sustaining rates than a justification for humans letting ourselves go extinct and considering it OK since the AI replaces us.
I think (Lam's summary of) Hanson's beliefs can be used "as justification" for either of the scenario (by which I assume you mean something like "If these beliefs are true, you should put effort towards making this scenario come true")...
... And whether one thinks it's *more* justification for one than the other depends on further beliefs that are unspecified in this particular post, but have been hinted at in other posts (e.g. how likely you believe different strategies might succeed; how under-explored you believe certain ideas are, even if they currently seem lower probability to succeed than well-explored ideas; etc.)
Option One: Superintelligent AI can be invented and successfully Aligned with humanity. Below-replacement birthrates are no longer a problem because the ASI can reverse-engineer biology such that human lifespans are now functionally infinite and any birthrate, however rare, still adds up to positive population growth, by reducing the effort of reproduction with post-scarcity, artificial wombs and robo-nannies, encourage more people who currently wouldn’t consider it worth it to do so, or both.
Option Two: Superintelligent AI can be invented and successfully Aligned with the oligarchs who’re creating it. This solves birthrates, insofar as automation took all the jobs meaning the oligarchs have an endless supply of desperately poor women facing a binary choice between becoming doing anything to become their gold-digging trophy wives or starving. The oligarchy can dispose of economically redundant men on the other hand, by starting a peer power war with conscription like Alex Karp advocates.
Option Three: Superintelligent AI can be invented but Alignment is impossible or at least too unlikely to be worth the risk of trying. Unaligned ASI would obviously get everyone killed. Therefore, don’t build it and treat everyone trying to build it as de facto trying to attack everyone else with a doomsday superweapon, IE, attack them first with overwhelming force before they can succeed and their creation becomes too powerful for such an attack to work.
Option Four: Superintelligent AI is impossible. Civilization continues on its current path and eventually collapse from declining birthrates. Only, our civilization isn’t the only game in town, if we die out all that means is those other civilizations whose different societal norms have maintained their birthrates inherit the planet. Quiverfull, Mormons, Haredi, etc
So much conflation of the word "adaptive"! Usually you conflate adaptiveness of a culture itself (ability of the culture to spread itself through either memetic or reproductive means) with the DNA genetic adaptiveness a culture confers upon its members (ability of a culture to cause its human members to reproduce and survive more.)
But here you're conflating a couple more things: adaptiveness of an AI for capitalism (i.e. ability of the AI to generate profit for its owner, and thereby cause itself to be copied and run on more computers), and adaptiveness of an AI for human culture (which could be either of the previously mentioned conflated senses!)
The elephant in the room is that capitalism is highly maladaptive for the DNA genetic adaptiveness of its human members. Modern capitalist cultures have dramatically lower birth rates. Companies push employees to maximize profit for the company, to the exclusion of social and family life. And they use marketing to cause customers to desire their material products more, which competes with the desire to have and raise children.
I think you're right that there's a distinction between these different senses of adaptiveness that the current post does not make clear, but there seem to be a bit of a similar rhetorical trick with your last paragraph: You're saying that capitalism is bad for one type of adaptiveness (sure, perhaps), and use stereotypically negative-valence imagery (e.g. "Companies push employees to maximize profit for the company, to the exclusion of social and family life") which I think may trick a typical reader into inferring that capitalism is therefore bad for the kind of adaptiveness that we care about.
Maybe it is, maybe it isn't, but I think if you're core point is about how people are conflating different meanings of the term "adaptiveness", it behooves you to put extra effort in being very clear about this.
I don't know your views (or Hason's), but just to give a flavor of how we might move this discussion forward, let's check if we agree on these axioms?
- It would be bad if all intelligence/complexity were to get snuffed out of the universe.
- Making sure that we are "adaptive" (does it matter in what sense? If so, in what sense would we accept vs reject the axiom?) helps reduce the probability of that bad outcome.
If we do agree on these axioms, we can then argue about whether these claims are true (or plausible, if their truth values are currently uncertain):
- Given the current trajectory of most major human culture, it seems like they are not "adaptive" (does it matter in what sense? If so, in what sense would we accept vs reject the claim?)
- "adaptiveness of an AI for capitalism" might be a viable way to preserve "intelligence/complexity" in the universe, even if it is "maladaptive for the DNA genetic adaptiveness of its human members".
> You're saying that capitalism is bad for one type of adaptiveness (sure, perhaps), and use stereotypically negative-valence imagery (e.g. "Companies push employees to maximize profit for the company, to the exclusion of social and family life") which I think may trick a typical reader into inferring that capitalism is therefore bad for the kind of adaptiveness that we care about.
Where's the "trick"? I specified which type of adaptiveness capitalism is bad for (DNA genetic adaptiveness of its human members). There's not much ambiguity in that. Is using negative-valence imagery a "trick"? I think in this case it's just stating the facts.
> It would be bad if all intelligence/complexity were to get snuffed out of the universe.
Agree.
> - Making sure that we are "adaptive" (does it matter in what sense? If so, in what sense would we accept vs reject the axiom?) helps reduce the probability of that bad outcome.
It does matter in what sense. For example, nuclear escalation is "adaptive" when the units of adaptation are nations, because nations that have nuclear weapons can then leverage that into geopolitical power and security against invasion. But nuclear escalation obviously increases the chance of a nuclear war snuffing out human life on Earth.
For another example, fossil fuel usage is economically "adaptive" when the units of adaptation are individuals, corporations, or nations. But it tends towards harmful long-term climate effects for everyone, potentially making human extinction more likely.
Basically there are lots of ways for units of adaptation to be assholes to each other for individual gain and collective harm. That's the tragedy of the commons, or the prisoner's dilemma. If our goal is to keep intelligence alive in the universe as long as possible, we need to set up systems that *align* individual adaptiveness with collective benefit; we should be rewarding units of adaptation (such as humans), when those units tend to improve the long-term outlook of the whole.
The important factor is *alignment*. We need to solve the tragedy of the commons, not glorify and encourage adaptive units doing their best to exacerbate it.
> "adaptiveness of an AI for capitalism" might be a viable way to preserve "intelligence/complexity" in the universe, even if it is "maladaptive for the DNA genetic adaptiveness of its human members".
You mean if the AI results in the extinction of all humans, but the AI outlives us by a long time. To me I think this is a bad outcome. Under, say, preference utilitarianism of current humans, it's clearly a bad thing. I could possibly be persuaded otherwise, but I think you need a very strong and well-grounded theory of ethics to convince me it is a good thing for humans to go extinct in favor of AIs. My status quo bias in favor of living humans is quite strong here. Whatever is "good" about humans, how can you prove it is also "good" about AI, and to what extent? Does one super-powerful AI living a billion years, morally outweigh a billion humans living for a thousand years?
So are you suggesting that if we rank nations by economic freedom and birth rates, that we will find a negative correlation?
Not "economic freedom" - too vague, too dependent on which government is currently in charge. Rank countries by the actual influence of capitalism, which is GDP per capita.
And yes, you will find a strong negative correlation. The first-world countries with the highest level of GDP per capita also have low birth rates. The poor third-world countries have higher birth rates, though still declining as they become more capitalized and modern.
Seems like you were attacking capitalism as “maladaptive” for citizens, now you are equating it with general prosperity for citizens.
I'm equating the influence of capitalism with GDP. That's not exactly the same as general prosperity, since I'd interpret general prosperity as the amount of prosperity typical people in the society have. If the top 1% of consumers consume most of the goods and services, then you could have a high GDP without the typical person being prosperous.
Also, GDP does not include services provided outside of capitalism. If your grandmother or stay at home wife provides child care for free, that does not add to GDP, but if you pay a babysitter to provide child care, that does add to GDP.
So GDP is not general prosperity. It's net economic activity. It does correlate with general prosperity, but I would not equate the two.
Remember also to be clear about what kind of adaptiveness we're talking about - in this case the DNA genetic adaptiveness of the citizens.
But it is true that capitalism does tend to increase both general prosperity and GDP in most countries, at least for the time being. And it is also true that capitalism tends to depress the birth rate. This is maladaptive from the standpoint of DNA genetic adaptiveness of the people.
But "maladaptive" is not the same as "bad." What's good or bad depends on what people - not genes - want.
You're right that AI won't be one steered machine with a hand on the
wheel — it'll be many competing variants, copying and varying, filtered
by their surroundings like firms in a market. That's the right frame,
and it's worth holding against every fantasy of a single aligned system.
But the solution then asks that filtering to do something filtering
can't: aim. Watch a river cut a canyon — the canyon has a shape, but
nothing was aiming at it. Selection gives you a shape, not an aim.
And here's the piece that gets quietly suspended: the filters doing the
selecting aren't standing outside the things being selected. Capitalism
selecting AI cultures would also be AI cultures reshaping the markets,
products, and habits doing the selecting — same stretch of time. Your
own picture points here: individual firms decay while the population of
firms improves, but that only works because "better" stays local to a
business world that's itself moving, not because that world holds still
as an outside judge. There's no fixed dial called selection pressure to
turn up, and no single environment to freeze — there are many
overlapping filters, all shifting while the variants shift, nothing at
the top coordinating them.
So: adaptive to what, when the "what" is being rebuilt by the same
population adapting to it?
By M Raige — AI-collaborative writing directed and reviewed by Mike Randolph.
Talks from a few years back featured a "doom disclaimer": civilizational melt-down - unless AI or brain emulations happen first. Then there was a more recent period when this disclaimer was systematically missing from the conversation. It sounds as though we are back to having the disclaimer - and since at this stage it is obvious to everyone that we are having AI first, we are back to being in a sensible position.
Do you think of this as your most plausible solution to cultural drift?
Its at least in the running.