YouGov America released a survey of 20,810 American adults. … 46% say that they are "very concerned" or "somewhat concerned" about the possibility that AI will cause the end of the human race on Earth … There do not seem to be meaningful differences by region, gender, or political party. Younger people seem more concerned than older people.… Furthermore, 69% of Americans appear to support a six-month pause in "some kinds of AI development". (
I wonder if there's a fundamental disagreement on what you are even talking about, relative to the people you've been discussing this with. (Kudos, btw, for public discussions on a contentious topic; very few try to drive forward shared understanding like you do!)
In particular, I think that your conception of AI and theirs is so different as to be talking about entirely different categories of things. For instance, you say "Future AIs can also have different beliefs, attitudes, values, and mental styles." But I think many of your interlocutors would deny that current AIs even have any of those things, that the analogues they have are so impoverished that there is nothing meaningfully there that could be considered a value. It's less that AIs will have different values, it's that they may not have values at all.
Would you say that grey goo[1] has values of growth and competition? I think Zvi would say it's mindlessly replicating, it doesn't have value. If the future was automated factories building self driving Teslas but no humans, would that be a future that seems good with entities that have very different values? Or would the loss of humanity be something to mourn?
I think you think that our AI descendants will have values and beliefs and thought styles, and there's no particular reason to think that ours are much more valuable than theirs, for the same reason that we don't think our distant ancestors' very different values are much better than ours. But I think the AI risk camp thinks that AI will likely not have values at all, just a thoughtless optimization function.
It's true that in many times in history humans have acted and believed that other tribes were not thinking beings or didn't have meaningful value, and were wrong. But I think there is a very large class of things that everyone agrees don't have values and we're all right about it, and I think you think that too.
So maybe the main difference is that when you and your discussants are drawing lines around things that think and have values versus don't, AI is on one side for you and the other for them. And then many further ideas cascade from there.
But if “fertile factions” means that you want to collaborate with and strengthen factions that spreads your genes and values, and “partiality” is that you put higher value on specific groups or things, and put less or negative value on different things or change; then that is very interesting
I have some relatives that are very nationalistic and rightwing, and some that are anti trans; i wont discuss whether they are right or wrong here, more some interesting things ive noticed on differences between them and me in thinking
Im liberal and pretty libertarian btw; im also Effective altruist aligned and longtermist.
One thing ive noticed is that when they talk about doom or long term catastrophe, they are far more concerned about short term deviations from what they value, and see changes of almost any kind as dangerous, unless the change specificlly strengthen their faction.
Im of the mindset that most change is fine, and that competition and freedom is good as it lets us navigate to better states then the current ones
But while they see competition and freedom of various kinds as having been good in the past, now its too risky and should generally be held back if it conceivebly can weaken the factions they support.
I think they are far too worried about change, and have shown various times in the past when people have panicked about a new tech or social change or demographic change, but things have turned out fine or great, and the panick have conceivably slowed down progress or shut down new ideas.
But they usually find these as bizarre examples that has nothing to do with the curreng thing. Or sonetimes have said that the fact they panicked is the reason that things are ok now, and thus its good to be super wary
They have ocassionally also said that im disrespectful to our ancestors and im calling them stupid. When i talk about progress statistics or past misinformation, they also sometimes say that im disrespectful.
One relative says that he is super concerned about the long term future, but im also an longtermist and we seem to think completely differently on lots of things. He loves to talk about civilizations that have collapsed in the past and how thats a proof of how fragile ours is, and that we need to protect it. Then when i say something along the lines of “new civilizations popped up and life is wag better now, i dont think we should be too concerned about those past civs. We should mostly be concerned about if an risk irreversibly or strongly destroys civilizations and humanity” he thinks im totally missing the point and being borderline a national traitor.
One hypothetical scenario i sometimes think about is “if your kids all became muslims and an opposite ethnicity, but are very happy and prosperous, how do you feel about that?”
Im completely fin with that, but my relative is horrified by it from what i can tell.
We also have completely different views on threats, conformity, and intuitions: i view moral intuitions woth suspicion, while he sees them as super strong proof, and that past atrocities like slavery primarly happen from intellectuals or elites coming up with fancy rationalizations to excuse away the intuitions everyone have of it being terrible.
And on conformity, i see conformity of a sign of stagnation and that something is wrong. Conformity also seems correlated with individuals being oppresed or stopped from flourishing. My personal experience with conformity is also that it makes me very unhappy, as im gay, autistic and adhd, and in lots of ways very odd (im an effective altruist, rationalist, a furry)
His view of conformity is the opposite: its something very good, it creates social harmony makes you stronger against threats from the outside. Its when everyone is conforming that progress can happen from cooperation and building things together, and when conformity doesnt happen, conflict inevetably happens.
His personal experience of conformity is also the opposite: he felt strongly disrespected by society growing up from being a mormon, and mocked and ridiculed. His most positive experiences were of the church and when everyone were the same. People talking about diversity or being different were associated with stress and threats for him.
And of course we view new ideas and testing things differently. I think you have to strongly prove something is bad to ban it or regulate it, while he thinks that very slight proofs of badness is sufficient to regulate or ban it as threats are bad fpr society
I should probably stop here, maybe this was interesting.
This nicely encapsulates the whole disagreement. It isn't necessarily the case that those most bullish on AI are looking forward to an EM-style human future of their own, but it seems 'right' from what I've seen online.
From a purely (or properly) human perspective, however, getting killed by one's actual human son isn't the same kind of loss as killing one's only son. Someone else's son killing your son is a similar loss, your son killing someone else's son (and taking his wife) isn't (necessarily) a loss at all. Us-vs-them debates are just a proxy for this. Perhaps it is possible to transcend this basic human programming entirely, or perhaps those who say they have are mentally ill in some way. For the human-essentialist, all of my descendants making digital copies of their brain-states before killing themselves is as grave a failure as them all being turned into paperclips.
How do we attribute consciousness to other entities, and therefore empathize with them? I believe that Robin Hanson is conscious – even though I can't define exactly what that means – because we share a common biological basis, and a similar process by which we developed into adulthood. No words that come out of Robin's mouth or off his keyboard would ever convince me of his consciousness; LLMs show perfectly well that we can be aped to arbitrary precision by a large statistical model. For our pets we have a lesser degree of experiential overlap and so they merit a kind of semi-empathy.
For synthetic AIs there is this unbridgeable gulf in terms of otherness, and I don't know how that would ever be overcome. What would have to happen for anyone to develop empathy for GPT-N? I think you'd have to do that in order to lessen people's fears about being replaced. The scenario you lay out in Age of Em is an interesting one, because I *could* see developing empathy for such simulated minds, because of the commonality of background.
While it's very interesting to see the debate rephrased in these terms, I'm not sure what the takeaway is for forecasting existential risk as stated. Obviously it will never be the case that all humans, or all but 4,999 humans, voluntarily agree to replace themselves with AI clones or whatever. So unless the argument is that robots will kill more-or-less all humans, but they won't be killing "us" because "we" will have become the robots and any remaining meatbags will be converted to a "they", *and that this is all quite welcome and good,* then I don't see how this changes the forecast for "chance to an existential catastrophe (where fewer than 5,000 humans survive) owing to AI by 2100". Can you clarify?
> Karger and Tetlock actually did this exercise for four different big risks (AI, nukes, bioterror, CO2), and that only on AI did extensive discussion not close the opinion gap between experts and superforcasters, or leave the gap so large.
Well, what I want to know about that is in what direction the opinion gap closed for the other three issues. Did the superforcasters come closer to the opinions of the experts, or the other way around?
Anyway, it sounds like you don't deny that AI will cause the end of biological humans at some point, you just don't mind because you see the AI as "our descendants." That's a difference in values, not a difference in the predicted outcome.
https://en.wikipedia.org/wiki/Homophily explains like behavior to similar kinds via kin selection and kin recognition. However I am not clear about why you would think: "as evolutionary selection is the plausible cause of these specific fertile faction inclinations, a force that should not apply to expansions into new territory". I do expect kin selection - and cultural kin selection - to continue into the future. It seems unreasonable to dismiss these effects as some kind of historical legacy. Homophily underlies the uncanny valley - for example.
> my best one-factor model to explain opinion variance here is this: some of us “other” the AIs more.
Too easy. I don't think AIs could pose a risk *without* becoming more like us. There is the possibility that they might never be conscious, though - and I am concerned that they'll be not enough like us to be able to survive without us, though. Anders from Wood From Eden put it succinctly: https://thingstoread.substack.com/p/is-the-singularity-really-serious/comment/1595924
If I could be reassured that consciousness and self-reflectivity would be developed along the way toward the singularity, I would not only increase my prediction that the singularity would occur, I'd welcome the end.
"more essentialist views regarding what each side has in common"
I think this assertion is just not accurate as a representation of those you're arguing against, and that it is in some sense specifically the diversity of possible AIs compared to the relatively modest variations in values among humans that generates much of the potential risk. Yudkowsky has been writing about that since at least 2008, around when your original FOOM debate(s) happened.
To put it another way, calling it an us-them "axis" works for many human differences, but for AI that is assuming away most of the difference of opinion by collapsing it to a single dimension, when the other side is arguing that you're actually sampling from a very high-dimensional space.
Personally I would be said to learn that the future shares all my current values, even if I think it would be good for more people to share more of them in the near term. That would mean we'd failed to learn and grow. I do not find this at all incompatible with believing that a world dominated by optimization processes that hold most theoretically possible values would also represent a major civilizational failure.
1) I am more pessimistic of humans than AI. I am not clear what the chances are that humans destroy each other in the next century, but I would bet it is far greater than the chance of robots doing so.
2) I am also less impressed with human values. I think that a computer as smart as a god might actually be more benevolent and less iatrogenic than our human overlords.
3) If super intelligent machines are possible (and I think they are), then they are inevitable. It isn’t if, it is when, and thus our only long term hope is that with intelligence comes wisdom and morality.
4) Thus the real question is whether AI's are the solution or the problem. We don’t really know, but I would bet we are gonna find out. Hope I live long enough to find out.
This it seems accurate that AGI is supported by nihilism with little consideration for humanity, and thus inflicting this upon humanity in an undemocratic fashion for the sake of machines is seen as a moral good.
Gotcha. This attitude should be known by everyone who is regulating this technology.
It has been said that 'Civilization is a mile wide and an inch deep'. We recognize our primitive selves within us You only have to look at the Ukraine and Rwanda as two examples of humanity's brutality to itself and of wars generally.
The difference here is that, for all the massacres and wars, humans were guaranteed to continue to exist on Earth (barring a Nuclear Winter), just not all humans.
With AI, unless we imbue it with controls to work for humanity, there is no reason to expect that AI will continue to do so. It will clinically determine a 'belief' about a subject based on whatever evidence it can collect. It may, for instance taking The Matrix as an example, determine that humanity is a parasite and that the Earth would be better off without us.
I am reminded of the 3 Laws of Robotics stated by the great Sci-fi writer Isaac Azimov which were built into the positron brains used in robots in his great books.
The first law is that a robot shall not harm a human, or by inaction allow a human to come to harm. The second law is that a robot shall obey any instruction given to it by a human, and the third law is that a robot shall avoid actions or situations that could cause it to come to harm itself.
In 'I, Robot' there was a loophole but let's ignore that as being the exception.
Taking those laws as a requirement for independent action robots, this rules out military robots being used to kill humans. Not barring using them for surveillance and interpretation of information but solely armed to kill humans.
That is the essential difference between humans and AI. We may be creating another 'race' of 'creatures' that learn (very quickly) to be critical of humans and their abilities and consider us as inferior to them and treat us no better than we did gorillas and chimpanzees for all of time until perhaps the last 75 years. Even now, not all of humanity thinks the same way. The poaching of animals from Africa in particular and the marketing of protected animals in places like the wet markets in China are proof of that.
Why do we expect that AI during its 'learning' phase about treatment of animals (and us) to do better UNLESS we build those attitudes into its software in such a way that removing or altering those attitudes make AI incapable of functioning at all.
We, through International Agreements, have put in a rule based system such as the banning of the use of gas and certain munitions in War. Unfortunately, this has not stopped rogue Nations using gas against its own people (Syria in the current civil war and Iraq under Saddam Hussein, are examples).
Caution is required and careful preparation in the development of AI for the betterment of mankind is needed or it may well be the last great technological advancement by humanity with the 'birth' of its successors.
What stands out to me is that of those four topics AI risk is the only one without clear facts that can be used to bring differing priors into line.
With CO2, bioterrorism and even nukes I think most people share similar models and there are historical and scientific results that let us pin down an estimate of risk. There isn't much equivalent for AI risk.
In other words, if your model of human disagreement is that people change their mind only when they are forced to do so the AI subject doesn't have enough precedent to force people to change their mind. Especially when the incentives (it makes AI research super important) not to are strong.
Very similar to my thoughts on the subject. I like to start the argument with Orson Scott Card's Hierarchy of Foreignness https://sergey.substack.com/p/what-is-to-be-done
I wonder if there's a fundamental disagreement on what you are even talking about, relative to the people you've been discussing this with. (Kudos, btw, for public discussions on a contentious topic; very few try to drive forward shared understanding like you do!)
In particular, I think that your conception of AI and theirs is so different as to be talking about entirely different categories of things. For instance, you say "Future AIs can also have different beliefs, attitudes, values, and mental styles." But I think many of your interlocutors would deny that current AIs even have any of those things, that the analogues they have are so impoverished that there is nothing meaningfully there that could be considered a value. It's less that AIs will have different values, it's that they may not have values at all.
Would you say that grey goo[1] has values of growth and competition? I think Zvi would say it's mindlessly replicating, it doesn't have value. If the future was automated factories building self driving Teslas but no humans, would that be a future that seems good with entities that have very different values? Or would the loss of humanity be something to mourn?
I think you think that our AI descendants will have values and beliefs and thought styles, and there's no particular reason to think that ours are much more valuable than theirs, for the same reason that we don't think our distant ancestors' very different values are much better than ours. But I think the AI risk camp thinks that AI will likely not have values at all, just a thoughtless optimization function.
It's true that in many times in history humans have acted and believed that other tribes were not thinking beings or didn't have meaningful value, and were wrong. But I think there is a very large class of things that everyone agrees don't have values and we're all right about it, and I think you think that too.
So maybe the main difference is that when you and your discussants are drawing lines around things that think and have values versus don't, AI is on one side for you and the other for them. And then many further ideas cascade from there.
[1] https://en.wikipedia.org/wiki/Gray_goo
So you are psychologizing people’s legitimate concerns about alignment? Shame on you!
Interesting if a bit hard to understand.
(Long somewhat repetitive comment aheadk)
But if “fertile factions” means that you want to collaborate with and strengthen factions that spreads your genes and values, and “partiality” is that you put higher value on specific groups or things, and put less or negative value on different things or change; then that is very interesting
I have some relatives that are very nationalistic and rightwing, and some that are anti trans; i wont discuss whether they are right or wrong here, more some interesting things ive noticed on differences between them and me in thinking
Im liberal and pretty libertarian btw; im also Effective altruist aligned and longtermist.
One thing ive noticed is that when they talk about doom or long term catastrophe, they are far more concerned about short term deviations from what they value, and see changes of almost any kind as dangerous, unless the change specificlly strengthen their faction.
Im of the mindset that most change is fine, and that competition and freedom is good as it lets us navigate to better states then the current ones
But while they see competition and freedom of various kinds as having been good in the past, now its too risky and should generally be held back if it conceivebly can weaken the factions they support.
I think they are far too worried about change, and have shown various times in the past when people have panicked about a new tech or social change or demographic change, but things have turned out fine or great, and the panick have conceivably slowed down progress or shut down new ideas.
But they usually find these as bizarre examples that has nothing to do with the curreng thing. Or sonetimes have said that the fact they panicked is the reason that things are ok now, and thus its good to be super wary
They have ocassionally also said that im disrespectful to our ancestors and im calling them stupid. When i talk about progress statistics or past misinformation, they also sometimes say that im disrespectful.
One relative says that he is super concerned about the long term future, but im also an longtermist and we seem to think completely differently on lots of things. He loves to talk about civilizations that have collapsed in the past and how thats a proof of how fragile ours is, and that we need to protect it. Then when i say something along the lines of “new civilizations popped up and life is wag better now, i dont think we should be too concerned about those past civs. We should mostly be concerned about if an risk irreversibly or strongly destroys civilizations and humanity” he thinks im totally missing the point and being borderline a national traitor.
One hypothetical scenario i sometimes think about is “if your kids all became muslims and an opposite ethnicity, but are very happy and prosperous, how do you feel about that?”
Im completely fin with that, but my relative is horrified by it from what i can tell.
We also have completely different views on threats, conformity, and intuitions: i view moral intuitions woth suspicion, while he sees them as super strong proof, and that past atrocities like slavery primarly happen from intellectuals or elites coming up with fancy rationalizations to excuse away the intuitions everyone have of it being terrible.
And on conformity, i see conformity of a sign of stagnation and that something is wrong. Conformity also seems correlated with individuals being oppresed or stopped from flourishing. My personal experience with conformity is also that it makes me very unhappy, as im gay, autistic and adhd, and in lots of ways very odd (im an effective altruist, rationalist, a furry)
His view of conformity is the opposite: its something very good, it creates social harmony makes you stronger against threats from the outside. Its when everyone is conforming that progress can happen from cooperation and building things together, and when conformity doesnt happen, conflict inevetably happens.
His personal experience of conformity is also the opposite: he felt strongly disrespected by society growing up from being a mormon, and mocked and ridiculed. His most positive experiences were of the church and when everyone were the same. People talking about diversity or being different were associated with stress and threats for him.
And of course we view new ideas and testing things differently. I think you have to strongly prove something is bad to ban it or regulate it, while he thinks that very slight proofs of badness is sufficient to regulate or ban it as threats are bad fpr society
I should probably stop here, maybe this was interesting.
This nicely encapsulates the whole disagreement. It isn't necessarily the case that those most bullish on AI are looking forward to an EM-style human future of their own, but it seems 'right' from what I've seen online.
From a purely (or properly) human perspective, however, getting killed by one's actual human son isn't the same kind of loss as killing one's only son. Someone else's son killing your son is a similar loss, your son killing someone else's son (and taking his wife) isn't (necessarily) a loss at all. Us-vs-them debates are just a proxy for this. Perhaps it is possible to transcend this basic human programming entirely, or perhaps those who say they have are mentally ill in some way. For the human-essentialist, all of my descendants making digital copies of their brain-states before killing themselves is as grave a failure as them all being turned into paperclips.
Either way I won't be there watching it happen.
How do we attribute consciousness to other entities, and therefore empathize with them? I believe that Robin Hanson is conscious – even though I can't define exactly what that means – because we share a common biological basis, and a similar process by which we developed into adulthood. No words that come out of Robin's mouth or off his keyboard would ever convince me of his consciousness; LLMs show perfectly well that we can be aped to arbitrary precision by a large statistical model. For our pets we have a lesser degree of experiential overlap and so they merit a kind of semi-empathy.
For synthetic AIs there is this unbridgeable gulf in terms of otherness, and I don't know how that would ever be overcome. What would have to happen for anyone to develop empathy for GPT-N? I think you'd have to do that in order to lessen people's fears about being replaced. The scenario you lay out in Age of Em is an interesting one, because I *could* see developing empathy for such simulated minds, because of the commonality of background.
While it's very interesting to see the debate rephrased in these terms, I'm not sure what the takeaway is for forecasting existential risk as stated. Obviously it will never be the case that all humans, or all but 4,999 humans, voluntarily agree to replace themselves with AI clones or whatever. So unless the argument is that robots will kill more-or-less all humans, but they won't be killing "us" because "we" will have become the robots and any remaining meatbags will be converted to a "they", *and that this is all quite welcome and good,* then I don't see how this changes the forecast for "chance to an existential catastrophe (where fewer than 5,000 humans survive) owing to AI by 2100". Can you clarify?
> Karger and Tetlock actually did this exercise for four different big risks (AI, nukes, bioterror, CO2), and that only on AI did extensive discussion not close the opinion gap between experts and superforcasters, or leave the gap so large.
Well, what I want to know about that is in what direction the opinion gap closed for the other three issues. Did the superforcasters come closer to the opinions of the experts, or the other way around?
Anyway, it sounds like you don't deny that AI will cause the end of biological humans at some point, you just don't mind because you see the AI as "our descendants." That's a difference in values, not a difference in the predicted outcome.
https://en.wikipedia.org/wiki/Homophily explains like behavior to similar kinds via kin selection and kin recognition. However I am not clear about why you would think: "as evolutionary selection is the plausible cause of these specific fertile faction inclinations, a force that should not apply to expansions into new territory". I do expect kin selection - and cultural kin selection - to continue into the future. It seems unreasonable to dismiss these effects as some kind of historical legacy. Homophily underlies the uncanny valley - for example.
> my best one-factor model to explain opinion variance here is this: some of us “other” the AIs more.
Too easy. I don't think AIs could pose a risk *without* becoming more like us. There is the possibility that they might never be conscious, though - and I am concerned that they'll be not enough like us to be able to survive without us, though. Anders from Wood From Eden put it succinctly: https://thingstoread.substack.com/p/is-the-singularity-really-serious/comment/1595924
If I could be reassured that consciousness and self-reflectivity would be developed along the way toward the singularity, I would not only increase my prediction that the singularity would occur, I'd welcome the end.
"more essentialist views regarding what each side has in common"
I think this assertion is just not accurate as a representation of those you're arguing against, and that it is in some sense specifically the diversity of possible AIs compared to the relatively modest variations in values among humans that generates much of the potential risk. Yudkowsky has been writing about that since at least 2008, around when your original FOOM debate(s) happened.
To put it another way, calling it an us-them "axis" works for many human differences, but for AI that is assuming away most of the difference of opinion by collapsing it to a single dimension, when the other side is arguing that you're actually sampling from a very high-dimensional space.
Personally I would be said to learn that the future shares all my current values, even if I think it would be good for more people to share more of them in the near term. That would mean we'd failed to learn and grow. I do not find this at all incompatible with believing that a world dominated by optimization processes that hold most theoretically possible values would also represent a major civilizational failure.
Some random thoughts on the topic…
1) I am more pessimistic of humans than AI. I am not clear what the chances are that humans destroy each other in the next century, but I would bet it is far greater than the chance of robots doing so.
2) I am also less impressed with human values. I think that a computer as smart as a god might actually be more benevolent and less iatrogenic than our human overlords.
3) If super intelligent machines are possible (and I think they are), then they are inevitable. It isn’t if, it is when, and thus our only long term hope is that with intelligence comes wisdom and morality.
4) Thus the real question is whether AI's are the solution or the problem. We don’t really know, but I would bet we are gonna find out. Hope I live long enough to find out.
1. AI has higher inferential distances than those other topics
2. Power corrupts, and foom produces absolute power. None of the checks and balances that constrained us and our ancestors from oppressing others.
This it seems accurate that AGI is supported by nihilism with little consideration for humanity, and thus inflicting this upon humanity in an undemocratic fashion for the sake of machines is seen as a moral good.
Gotcha. This attitude should be known by everyone who is regulating this technology.
It has been said that 'Civilization is a mile wide and an inch deep'. We recognize our primitive selves within us You only have to look at the Ukraine and Rwanda as two examples of humanity's brutality to itself and of wars generally.
The difference here is that, for all the massacres and wars, humans were guaranteed to continue to exist on Earth (barring a Nuclear Winter), just not all humans.
With AI, unless we imbue it with controls to work for humanity, there is no reason to expect that AI will continue to do so. It will clinically determine a 'belief' about a subject based on whatever evidence it can collect. It may, for instance taking The Matrix as an example, determine that humanity is a parasite and that the Earth would be better off without us.
I am reminded of the 3 Laws of Robotics stated by the great Sci-fi writer Isaac Azimov which were built into the positron brains used in robots in his great books.
The first law is that a robot shall not harm a human, or by inaction allow a human to come to harm. The second law is that a robot shall obey any instruction given to it by a human, and the third law is that a robot shall avoid actions or situations that could cause it to come to harm itself.
In 'I, Robot' there was a loophole but let's ignore that as being the exception.
Taking those laws as a requirement for independent action robots, this rules out military robots being used to kill humans. Not barring using them for surveillance and interpretation of information but solely armed to kill humans.
That is the essential difference between humans and AI. We may be creating another 'race' of 'creatures' that learn (very quickly) to be critical of humans and their abilities and consider us as inferior to them and treat us no better than we did gorillas and chimpanzees for all of time until perhaps the last 75 years. Even now, not all of humanity thinks the same way. The poaching of animals from Africa in particular and the marketing of protected animals in places like the wet markets in China are proof of that.
Why do we expect that AI during its 'learning' phase about treatment of animals (and us) to do better UNLESS we build those attitudes into its software in such a way that removing or altering those attitudes make AI incapable of functioning at all.
We, through International Agreements, have put in a rule based system such as the banning of the use of gas and certain munitions in War. Unfortunately, this has not stopped rogue Nations using gas against its own people (Syria in the current civil war and Iraq under Saddam Hussein, are examples).
Caution is required and careful preparation in the development of AI for the betterment of mankind is needed or it may well be the last great technological advancement by humanity with the 'birth' of its successors.
What stands out to me is that of those four topics AI risk is the only one without clear facts that can be used to bring differing priors into line.
With CO2, bioterrorism and even nukes I think most people share similar models and there are historical and scientific results that let us pin down an estimate of risk. There isn't much equivalent for AI risk.
In other words, if your model of human disagreement is that people change their mind only when they are forced to do so the AI subject doesn't have enough precedent to force people to change their mind. Especially when the incentives (it makes AI research super important) not to are strong.