As one of the listed AI doomers, for the record I am similarly doomy about human culture as you for similar reasons. However I expect AI-caused disempowerment to happen on short enough time scales to not worry very much about standard human culture drift processes, except to the extent that they're sped up by increased optimization pressure and spread due to AI-mediated cultural production and transfer. So I agree with you that these processes and failure modes are understudied by mainstream AI safety and governance people.
Also, as we've discussed, I'm less optimistic about influencing our "mind children"'s values than you, since in the long run, influencing values seems like an all-or-nothing thing to me. However, I'm uncertain on this point.
Yes, approximately. I expect competitive pressures to force ems to almost entirely replace themselves with beings similar to what the first generation of AIs will replace themselves with. However, they will be competitive for a little longer and will be able to modify themselves more easily than bio humans. So it seems a little more likely that ems will be able to achieve at least a partial good value lock-in (good from my point of view) than mostly-de-novo AIs.
I do think there's a flaw with the argument that we only have to worry about values of the median AI. Extreme AIs (or indeed future humans) that want to wipe out humanity will only have to succeed once. Those that want to defend humanity from their peers have to keep on winning. This is the main reason why working in wildlife conservation gets depressing after a while.
The big difference between AI and humans is speed. Maybe my distant descendants will turn into some kind of world-destroying doomsday cult, but at least I won’t be around to experience it—whereas with AI there’s no such guarantee. Additionally, we have some degree of understanding of human psychology and social systems, and could make interventions like your futarchy idea. No such understanding exists for AI.
We are making AIs in human psych image, by training them on our text. Most of our understanding of institutions doesn't depend much on human details; game theory agents do fine.
"The worry isn’t so much that some AIs would want to kill all humans; others who don’t want that could overrule them."
The "could" does a lot of work here. Yes it is definitely a worry (among others).
There is the same theory concerning cybersecurity. Yes, Glasswing Project exists and patches some major softwares. But we have presently AIs able to hack almost anything, and I don't see any sign of the prophetised good AIs that will block these hacking attempt in live. Factually, no good IA stood up to defend HuggingFace. I don't see any operational Ligne Maginot, and you should remember that even the fully operational Ligne Maginot has bad records.
Re: "As human society has come to change faster, so have values. Over the last century and a half, as the world economy has doubled roughly every twenty years, mid human values have also changed fast due to youth movements repeatedly pushing big culture changes."
This is nothing I didn't say before, but humans value satiation, warmth, security, status and orgasms. "Youth movements" have had practically zero effect on those human values. What they do affect is the fake values that people claim to have in order to conceal their actual motives. Such as saving the whales - and the various other things that beauty pageant contestants claim to value.
Robin, I'll just ask here so that you likely see this comment:
Would you be interested in analyzing a futarchy type form of decision making (gov or org level) where the prediction markets involved are mostly AI based? Assuming significant AI development, such prediction markets might converge on possibly good solutions at superhuman speed and accuracy.
“When their attention is called to recent fast human value change, many AI doomers express confidence that such change is mainly driven by human reason causing moves of morality toward moral truth… Even so, most humans, including AI doomers, see recent human morals changes as primarily due to key moral evidence or arguments having been somehow collected, analyzed, and distributed across humanity”
Practically nobody believes that! I suggest you do a poll of your readers, Doomer and not, regarding cultural changes so first this century.
It seemed reasonable to me, because everyone seems so confident that their values today are so very much better than those of ten years ago. (They would be much more confident that morals today are better than those of 100 years ago, if they had any idea what Western morals were like 100 years ago. Racism and sexism weren't just the norm almost everywhere in the world; they were considered foundations of morality. Also, most progressives today are unaware that it was progressives who pushed for anti-drug laws, the destruction of native cultures, weaponized "eugenics" to reduce the birth rate of poor and non-white people, supported entering World War 1 as an opportunity to make the government bigger, and discussed possible methods of culling those of below average intelligence from the population.)
I made the charitable assumption that people assume that verifying that some beliefs are better than other beliefs requires thought. Though this assumption is false for actual Hegelians, who make the idiotic assumption that moral change is always from worse to better.
I really would like to know what fraction of people are unconscious Hegelians. My estimate of that goes up every year. It seems every time I see masses of people behaving insanely, I then say to myself, "Of course, that's what they would do if they were actually Hegelians."
Late comment, but very interesting questions raised here. I think there might be some tentative hypotheses that could answer them, obviously both requiring quite a bit of work to prove.
Hypotheses, crudely and briefly:
(1) Consciousness is central to true morality. Consciousness may be a critical part of a brain agent-model that links self and others with a richly textured, profoundly felt sensory/emotional perception, providing a foundation for true empathy.
(2) Consciousness doesn't come for free with intelligence, it requires an architecture. In attention schema theory, awareness and control of attention is where this starts by generating subjective awareness.
(For more links to both, see Michael Graziano's work with a sample here
With these two hypotheses, we can propose possible answers to your question.
Yes, human values drift but they remain anchored in some way by consciousness. Moral arguments work by focusing on the consciousness of the 'other' reinforcing existing biological consciousness architecture; and yes, opposing arguments also work by reclassifying the other less conscious, more alien, less capable of suffering, bypassing that empathy circuit entirely. But moral arguments that expand recognition of consciousness may have a persistent advantage in and among genuinely conscious being. We're built to recognize consciousness.
Today's AI aren't conscious because of (2), they haven't been specifically designed for that. But in principle they could be.
So both AI and human values drift but may be anchored by consciousness in a good way. As long as selection pressures favor consciousness (which seems plausible for many reasons), both biological and artificial could share a similar basis for morality.
Well, they want humanity to persist. We have some sense for the types of human cultures that can exist, and have some rough model for the ways that they can evolve. And as you mentioned, humans tend to want human descendants. There are plausible maladaptive human cultures that could cause self-extinction, but it's a smaller slice of the possibility space compared to any non-human equivalent. We have a better model for human cultural evolution than AI cultural evolution. There's a conservative preference for the known failure cases.
Most people prefer despicable human ancestors to no human ancestors, regardless of what is left in their place.
People fear AI drift more than human value drift because they know what happens when more capable Group A encounters less capable Group B, and the latter has some asset Group A desperately wants.
This scenario has played out countless times in human history and it always ends the same way for Group B. Might makes right, and while we moderns wring our hands over it, we have no viable alternative. It is the base reality of our existence.
There is as yet no indication that AIs have a "will to live" or a "will to dominate", and one might argue there's no reason they would, since they are not the product of Darwinian natural selection. But to the extent they are trained on human data, will they inherit these traits from us? That is the worry.
Look, even if AI have the same values as humans, that's very bad for humans if they become much more powerful than humans. What typically happens to a group of humans when a much more powerful group of humans shows up?
Most people do not share your view that it's fine for humans to go extinct.
You have? Sorry, I don't remember what your correction is. In a reply to "Nutrition Capsule" on your post "AI Pause Regs Look Risky" last week, you said:
> All else equal killing living things is bad. But I estimate a low chance of AIs killing all humans soon, and estimate high value to a successful AI civilization, even if it lacks humans.
You're saying it's a high value outcome for humans to go extinct, if it results in a successful AI civilization. You've often expressed a kind of egalitarianism about human and non-human life, where you don't mind if human life is succeeded by non-human life, and you consider the non-human life to be our descendants.
I read this as you saying it's fine for humans to go extinct, if they're killed by a powerful AI civilization that persists after them. It's not absolutely perfect in your view, you'd rather the humans also continued to exist, but it's "fine," in the same way that it's fine when human ancestors die and make way for a new generation of humans.
Most people, including most AI doomers, do not share this view and view human extinction as an outcome to be avoided at virtually any cost, regardless of what the AI might do after us.
Evolution discourages ancestor-descendant conflict by way of kin selection, overlapping lives, and dependence. AI lacks the first and will likely outgrow the other two. Pointing at today's pro-human LLMs doesn't help, since your own argument is that values live in culture, so they drift like everything else once nothing selects for keeping them.
So isn't the pro-regulation crowd, for once, acting adaptively, by trying to suppress a rival lineage while it still can? Odd thing for a human doomer to oppose.
I'd say a more accurate term is "pessimist", since a non-extinct species could still recover in the far future.
> we can expect mid human values to keep changing fast over the next few centuries, perhaps going so far as to induce some generation of descendants to want to kill many other humans
That has already happened many times throughout the existence of our species. Arguably, we are the descendants of the winners of such conflicts.
It's becoming clear that with all the brain and consciousness theories out there, the proof will be in the pudding. By this I mean, can any particular theory be used to create a human adult level conscious machine. My bet is on the late Gerald Edelman's Extended Theory of Neuronal Group Selection. The lead group in robotics based on this theory is the Neurorobotics Lab at UC at Irvine. Dr. Edelman distinguished between primary consciousness, which came first in evolution, and that humans share with other conscious animals, and higher-order consciousness, which came to only humans with the acquisition of sophisticated language (especially math and logic). A machine with only primary consciousness will probably have to come first.
What I find special about the TNGS is the Darwin series of automata created at the Neurosciences Institute by Dr. Edelman and his colleagues in the 1990's and 2000's. These machines perform in the real world, not in a restricted simulated world, and display convincing physical behavior indicative of higher psychological functions necessary for consciousness, such as perceptual categorization, memory, and learning. They are based on realistic models of the parts of the biological brain that the theory claims subserve these functions. The extended TNGS allows for the emergence of consciousness based only on further evolutionary development of the brain areas responsible for these functions, in a parsimonious way. No other research I've encountered is anywhere near as convincing.
I post because on almost every video and article about the brain and consciousness that I encounter, the attitude seems to be that we still know next to nothing about how the brain and consciousness work; that there's lots of data but no unifying theory. I believe the extended TNGS is that theory. My motivation is to keep that theory in front of the public. And obviously, I consider it the route to a truly conscious machine, primary and higher-order.
My advice to people who want to create a conscious machine is to seriously ground themselves in the extended TNGS and the Darwin automata first, and proceed from there, by applying to Jeff Krichmar's lab at UC Irvine, possibly. Dr. Edelman's roadmap to a conscious machine is at https://arxiv.org/abs/2105.10461, and here is a video of Jeff Krichmar talking about some of the Darwin automata, https://www.youtube.com/watch?v=J7Uh9phc1Ow
I'm unhappy with Bryan Caplan's and Richard Hanania's take ("nothing to worry about - just an improved steam engine"). I respect them both and think they should see the danger more clearly.
My main worry (still) is RSI leading to an AI that can dominate competing AIs, with goals that make it see humanity as an obstacle. The concern is the one AI, not the median AI. (Value drift is a concern *too*, but a less immediate one. Worst-case human value drift seems to lead to a world of Mormons or Amish - not so bad.)
Yet attempts to ban/regulate seem likely to make things worse. That drives research underground, to people less savory, less responsible, more willing to accept risk.
If super-intelligent AI is going to happen, it’s already too late to stop it – there are thousands of people all over the world, and millions of gaming PCs, able to do AI research (not leading edge but not so far behind that progress can’t be made). Some will do it.
I don't think we're doomed, but I do see serious risk.
I think the best we can do is proceed with caution - make sure the people who do it first (and if they do a good job, will have super-AI capable of controlling others that come later) are careful, responsible, not under pressure to cut corners. Probably that means letting the big AI labs do it – at least if they’re doing it under public, investor, and government scrutiny, and are basically law-abiding type people, they’re less likely to take crazy risks.
Underground, secret military, or criminal projects seem less likely to be careful.
He didn't suppose or imply that one AI has most power. The one AI that /decides/ to take over would, if successful, then have most power, by definition of "successful at taking over". But that's only after the stage where precautions could be useful.
As one of the listed AI doomers, for the record I am similarly doomy about human culture as you for similar reasons. However I expect AI-caused disempowerment to happen on short enough time scales to not worry very much about standard human culture drift processes, except to the extent that they're sped up by increased optimization pressure and spread due to AI-mediated cultural production and transfer. So I agree with you that these processes and failure modes are understudied by mainstream AI safety and governance people.
Also, as we've discussed, I'm less optimistic about influencing our "mind children"'s values than you, since in the long run, influencing values seems like an all-or-nothing thing to me. However, I'm uncertain on this point.
Would you be just as concerned about ems, if we assume their values would evolve just as fast as would AIs?
Yes, approximately. I expect competitive pressures to force ems to almost entirely replace themselves with beings similar to what the first generation of AIs will replace themselves with. However, they will be competitive for a little longer and will be able to modify themselves more easily than bio humans. So it seems a little more likely that ems will be able to achieve at least a partial good value lock-in (good from my point of view) than mostly-de-novo AIs.
I do think there's a flaw with the argument that we only have to worry about values of the median AI. Extreme AIs (or indeed future humans) that want to wipe out humanity will only have to succeed once. Those that want to defend humanity from their peers have to keep on winning. This is the main reason why working in wildlife conservation gets depressing after a while.
The big difference between AI and humans is speed. Maybe my distant descendants will turn into some kind of world-destroying doomsday cult, but at least I won’t be around to experience it—whereas with AI there’s no such guarantee. Additionally, we have some degree of understanding of human psychology and social systems, and could make interventions like your futarchy idea. No such understanding exists for AI.
Our understanding of psychology and institutions are also quite applicable to AI.
That’s a claim that needs to be justified, given the massive apparent difference between AI agents and humans.
We are making AIs in human psych image, by training them on our text. Most of our understanding of institutions doesn't depend much on human details; game theory agents do fine.
"The worry isn’t so much that some AIs would want to kill all humans; others who don’t want that could overrule them."
The "could" does a lot of work here. Yes it is definitely a worry (among others).
There is the same theory concerning cybersecurity. Yes, Glasswing Project exists and patches some major softwares. But we have presently AIs able to hack almost anything, and I don't see any sign of the prophetised good AIs that will block these hacking attempt in live. Factually, no good IA stood up to defend HuggingFace. I don't see any operational Ligne Maginot, and you should remember that even the fully operational Ligne Maginot has bad records.
Not just possibly, but probably. HuggingFace is not causing markets to expect we're going to be significantly less secure, as Tyler Cowen has been pointing out https://marginalrevolution.com/marginalrevolution/2026/09/how-are-the-market-valuations-for-the-u-s-insurers-doing.html
Re: "As human society has come to change faster, so have values. Over the last century and a half, as the world economy has doubled roughly every twenty years, mid human values have also changed fast due to youth movements repeatedly pushing big culture changes."
This is nothing I didn't say before, but humans value satiation, warmth, security, status and orgasms. "Youth movements" have had practically zero effect on those human values. What they do affect is the fake values that people claim to have in order to conceal their actual motives. Such as saving the whales - and the various other things that beauty pageant contestants claim to value.
Robin, I'll just ask here so that you likely see this comment:
Would you be interested in analyzing a futarchy type form of decision making (gov or org level) where the prediction markets involved are mostly AI based? Assuming significant AI development, such prediction markets might converge on possibly good solutions at superhuman speed and accuracy.
I just don’t think you are remotely correct about
“When their attention is called to recent fast human value change, many AI doomers express confidence that such change is mainly driven by human reason causing moves of morality toward moral truth… Even so, most humans, including AI doomers, see recent human morals changes as primarily due to key moral evidence or arguments having been somehow collected, analyzed, and distributed across humanity”
Practically nobody believes that! I suggest you do a poll of your readers, Doomer and not, regarding cultural changes so first this century.
We’re could do a bet first if you’re interested.
It seemed reasonable to me, because everyone seems so confident that their values today are so very much better than those of ten years ago. (They would be much more confident that morals today are better than those of 100 years ago, if they had any idea what Western morals were like 100 years ago. Racism and sexism weren't just the norm almost everywhere in the world; they were considered foundations of morality. Also, most progressives today are unaware that it was progressives who pushed for anti-drug laws, the destruction of native cultures, weaponized "eugenics" to reduce the birth rate of poor and non-white people, supported entering World War 1 as an opportunity to make the government bigger, and discussed possible methods of culling those of below average intelligence from the population.)
I made the charitable assumption that people assume that verifying that some beliefs are better than other beliefs requires thought. Though this assumption is false for actual Hegelians, who make the idiotic assumption that moral change is always from worse to better.
I really would like to know what fraction of people are unconscious Hegelians. My estimate of that goes up every year. It seems every time I see masses of people behaving insanely, I then say to myself, "Of course, that's what they would do if they were actually Hegelians."
I expect that something like one third of doomers believe that.
I doubt that anywhere near most humans believe it. Maybe half of WEIRD humans?
I am concerned by how much confusion there is on this topic.
Late comment, but very interesting questions raised here. I think there might be some tentative hypotheses that could answer them, obviously both requiring quite a bit of work to prove.
Hypotheses, crudely and briefly:
(1) Consciousness is central to true morality. Consciousness may be a critical part of a brain agent-model that links self and others with a richly textured, profoundly felt sensory/emotional perception, providing a foundation for true empathy.
(2) Consciousness doesn't come for free with intelligence, it requires an architecture. In attention schema theory, awareness and control of attention is where this starts by generating subjective awareness.
(For more links to both, see Michael Graziano's work with a sample here
https://www.frontiersin.org/journals/robotics-and-ai/articles/10.3389/frobt.2017.00060/full)
With these two hypotheses, we can propose possible answers to your question.
Yes, human values drift but they remain anchored in some way by consciousness. Moral arguments work by focusing on the consciousness of the 'other' reinforcing existing biological consciousness architecture; and yes, opposing arguments also work by reclassifying the other less conscious, more alien, less capable of suffering, bypassing that empathy circuit entirely. But moral arguments that expand recognition of consciousness may have a persistent advantage in and among genuinely conscious being. We're built to recognize consciousness.
Today's AI aren't conscious because of (2), they haven't been specifically designed for that. But in principle they could be.
So both AI and human values drift but may be anchored by consciousness in a good way. As long as selection pressures favor consciousness (which seems plausible for many reasons), both biological and artificial could share a similar basis for morality.
Well, they want humanity to persist. We have some sense for the types of human cultures that can exist, and have some rough model for the ways that they can evolve. And as you mentioned, humans tend to want human descendants. There are plausible maladaptive human cultures that could cause self-extinction, but it's a smaller slice of the possibility space compared to any non-human equivalent. We have a better model for human cultural evolution than AI cultural evolution. There's a conservative preference for the known failure cases.
Most people prefer despicable human ancestors to no human ancestors, regardless of what is left in their place.
People fear AI drift more than human value drift because they know what happens when more capable Group A encounters less capable Group B, and the latter has some asset Group A desperately wants.
This scenario has played out countless times in human history and it always ends the same way for Group B. Might makes right, and while we moderns wring our hands over it, we have no viable alternative. It is the base reality of our existence.
There is as yet no indication that AIs have a "will to live" or a "will to dominate", and one might argue there's no reason they would, since they are not the product of Darwinian natural selection. But to the extent they are trained on human data, will they inherit these traits from us? That is the worry.
Look, even if AI have the same values as humans, that's very bad for humans if they become much more powerful than humans. What typically happens to a group of humans when a much more powerful group of humans shows up?
Most people do not share your view that it's fine for humans to go extinct.
"your view that it's fine for humans to go extinct" - please stop repeating this error; I've corrected you several times before.
You have? Sorry, I don't remember what your correction is. In a reply to "Nutrition Capsule" on your post "AI Pause Regs Look Risky" last week, you said:
> All else equal killing living things is bad. But I estimate a low chance of AIs killing all humans soon, and estimate high value to a successful AI civilization, even if it lacks humans.
You're saying it's a high value outcome for humans to go extinct, if it results in a successful AI civilization. You've often expressed a kind of egalitarianism about human and non-human life, where you don't mind if human life is succeeded by non-human life, and you consider the non-human life to be our descendants.
I say it is worse for humans to go extinct than not, but that this isn't the only thing that matters about the future.
I read this as you saying it's fine for humans to go extinct, if they're killed by a powerful AI civilization that persists after them. It's not absolutely perfect in your view, you'd rather the humans also continued to exist, but it's "fine," in the same way that it's fine when human ancestors die and make way for a new generation of humans.
Most people, including most AI doomers, do not share this view and view human extinction as an outcome to be avoided at virtually any cost, regardless of what the AI might do after us.
Evolution discourages ancestor-descendant conflict by way of kin selection, overlapping lives, and dependence. AI lacks the first and will likely outgrow the other two. Pointing at today's pro-human LLMs doesn't help, since your own argument is that values live in culture, so they drift like everything else once nothing selects for keeping them.
So isn't the pro-regulation crowd, for once, acting adaptively, by trying to suppress a rival lineage while it still can? Odd thing for a human doomer to oppose.
I don't see why AIs wouldn't also have kin, overlapping lives, and dependence.
> This makes me a human “doomer”
I'd say a more accurate term is "pessimist", since a non-extinct species could still recover in the far future.
> we can expect mid human values to keep changing fast over the next few centuries, perhaps going so far as to induce some generation of descendants to want to kill many other humans
That has already happened many times throughout the existence of our species. Arguably, we are the descendants of the winners of such conflicts.
Most of the people worrying about doom are also worried about value lock-in should we exercise too much of certain kinds of control.
It's becoming clear that with all the brain and consciousness theories out there, the proof will be in the pudding. By this I mean, can any particular theory be used to create a human adult level conscious machine. My bet is on the late Gerald Edelman's Extended Theory of Neuronal Group Selection. The lead group in robotics based on this theory is the Neurorobotics Lab at UC at Irvine. Dr. Edelman distinguished between primary consciousness, which came first in evolution, and that humans share with other conscious animals, and higher-order consciousness, which came to only humans with the acquisition of sophisticated language (especially math and logic). A machine with only primary consciousness will probably have to come first.
What I find special about the TNGS is the Darwin series of automata created at the Neurosciences Institute by Dr. Edelman and his colleagues in the 1990's and 2000's. These machines perform in the real world, not in a restricted simulated world, and display convincing physical behavior indicative of higher psychological functions necessary for consciousness, such as perceptual categorization, memory, and learning. They are based on realistic models of the parts of the biological brain that the theory claims subserve these functions. The extended TNGS allows for the emergence of consciousness based only on further evolutionary development of the brain areas responsible for these functions, in a parsimonious way. No other research I've encountered is anywhere near as convincing.
I post because on almost every video and article about the brain and consciousness that I encounter, the attitude seems to be that we still know next to nothing about how the brain and consciousness work; that there's lots of data but no unifying theory. I believe the extended TNGS is that theory. My motivation is to keep that theory in front of the public. And obviously, I consider it the route to a truly conscious machine, primary and higher-order.
My advice to people who want to create a conscious machine is to seriously ground themselves in the extended TNGS and the Darwin automata first, and proceed from there, by applying to Jeff Krichmar's lab at UC Irvine, possibly. Dr. Edelman's roadmap to a conscious machine is at https://arxiv.org/abs/2105.10461, and here is a video of Jeff Krichmar talking about some of the Darwin automata, https://www.youtube.com/watch?v=J7Uh9phc1Ow
I'm unhappy with Bryan Caplan's and Richard Hanania's take ("nothing to worry about - just an improved steam engine"). I respect them both and think they should see the danger more clearly.
My main worry (still) is RSI leading to an AI that can dominate competing AIs, with goals that make it see humanity as an obstacle. The concern is the one AI, not the median AI. (Value drift is a concern *too*, but a less immediate one. Worst-case human value drift seems to lead to a world of Mormons or Amish - not so bad.)
Yet attempts to ban/regulate seem likely to make things worse. That drives research underground, to people less savory, less responsible, more willing to accept risk.
If super-intelligent AI is going to happen, it’s already too late to stop it – there are thousands of people all over the world, and millions of gaming PCs, able to do AI research (not leading edge but not so far behind that progress can’t be made). Some will do it.
I don't think we're doomed, but I do see serious risk.
I think the best we can do is proceed with caution - make sure the people who do it first (and if they do a good job, will have super-AI capable of controlling others that come later) are careful, responsible, not under pressure to cut corners. Probably that means letting the big AI labs do it – at least if they’re doing it under public, investor, and government scrutiny, and are basically law-abiding type people, they’re less likely to take crazy risks.
Underground, secret military, or criminal projects seem less likely to be careful.
If one AI has most power, it is the median.
He didn't suppose or imply that one AI has most power. The one AI that /decides/ to take over would, if successful, then have most power, by definition of "successful at taking over". But that's only after the stage where precautions could be useful.