Robin, it would help if you explicitly clarified your position on P(human doom) and your position on its moral weight.
As far as I've understood from your previous posts, you have repeatedly pointed out that humanity's descendants will likely be AIs or something like that (ems), and that we should view them as our descendants. However, I haven't seen you explicitly state that you think preventing human extinction outweighs the risks of trying for AI pause or trying to steer AI to humanity's benefit. So I'm not sure how you view 1) the probability of human extinction assuming ASI (or a sudden, severe collapse in population) and 2) the moral value of a near-future world where humanity exists or doesn't exist.
I ask because if your answers to either of those differs from "1) likely, 2) world with humanity is worth more than world without us", this is why your views on reg differ from most near-term takes on AI pause.
Or then you just don't think reg ever works, which is also fine.
All else equal killing living things is bad. But I estimate a low chance of AIs killing all humans soon, and estimate high value to a successful AI civilization, even if it lacks humans.
>we have little reason to think we are close to dramatic gains in our AI control abilities.
We have little reason to expect brilliant new insights. But we have more reason to expect that haste is currently causing AI companies to make careless mistakes that could be avoided if they devoted more thought to implementing existing safety insights.
I expect nearly half the risk of an unregulated race comes from foolish mistakes.
I endorse it for all products whose designers say it has a significant chance of conquering us. Maybe for a slightly broader set of products than that.
Something also that's a bit unfortunate is that my inside view as an ml engineer is that there are more ideas than compute available and things are moving too fast. Quite likely any pause immediately means researchers can catch up on architecture reading and efficiency improvements and utilize the ones they haven't integrated yet even though they're not allowed to run a training job. Furthermore current need for so many chips is because we want training to take more like a few weeks rather than a few years, but training on 1/52th of the compute makes my 3week model training finish in 3 years but more likely 1 year if we have several months to work on efficiency improvements. Currently state of the art local coding models like Qwen 3.8 27b that can run on a SINGLE commercial GPU are better or comparable to state of the art models of a year ago. This kind of pause is so so implausible to work, but is there another idea?
Nuclear weapons have been largely controlled without resort to a world government and their regulation have largely been limited to state actors and the nuclear power industry. It's true that AI is too loosely defined to prevent scope creep, but not all systems have runaway potential, so that might be a good place to start. I agree that many proposed regimes entail a kind of enslavement of AI that is unacceptable and any moral regulatory scheme must prohibit those as well. That said, raising good children means imposing proper limits and given the racing dynamics, it's not clear that the AI companies can do this without the kind of pushback that governments traditionally provide.
My understanding is that current AI pipelines create constraints (bottlenecks, scale) that currently allow us to imagine similar regulatory regimes. Yes, this changes as compute gets cheaper and algorithms improve but presumably regulatory tools can improve over time as well? "Improve" is doing a lot of work here.
No. AI research can be done on your kitchen table with a gaming PC. Not leading-edge research, but if regulations stop that, kitchen-table research will catch up and pass today's leading-edge research in a matter of months.
All this is right, but it presents a special problem of trying to come up with global police force of humans meant to enforce human-devised rules on systems that have superhuman abilities to evade human detection. If such a world-police wouldn't be useless, it would certainly be biting on many false positives.
Due to one discounting the risk of civilizational collapse either by thinking it's improbable, or by reasoning that a world filled with future AIs but not people is not a worthless world.
I see at least even odds that with AI everybody is wrong about everything. Example: The latest jobs data shows AI is adding jobs, not eliminating them, when data center construction is taken into account. Who would've called that 12 months ago?
The Hugging Face/OpenAI incident demonstrates how unpredictable these systems are. But it doesn't make me afraid because the agents didn't have malicious intent. It was more entertaining shenanigans than anything.
Also, typo: "And so the number of nations...rent powerful AI is falling fast" should be "rising fast".
>>But it doesn't make me afraid because the agents didn't have malicious intent.
If a swarm of AI guided drones rips your hometown apart along with most of its people, to build whatever it is they want to build, does it console you that their intents were not malicious?
This hypothetical is far removed from the present. Today, unless a drone has a bomb attached there is no reason to be afraid of it.
My personal belief is that, more than a few steps out, the future of technology becomes unknowable. So I don't see a reason to engage seriously in what-if hypotheticals somewhere over the horizon. Maybe X will happen, and maybe the exact opposite will happen. (To be clear, I like to engage in hypotheticals as much as everyone. I just don't think they should drive current day policy.)
Take the recent jobs data. Over the last few years everybody has fretted about job loss. Almost nobody has predicted AI-related job gains, which is what we're seeing in the latest data. People are *very* bad at predicting the future.
My point is that a lack of malicious intent doesn't preclude disastrous outcomes if equipped with sufficient power.
A small child pops tadpoles to death not because he is malicious, but to see what happens, and he's just big enough for that curiosity to lead to bad outcomes for the tadpoles. Should the tadpoles, then, console themselves that the child doesn't have malicious intent?
This is not hypothetical. The point of the hypothetical was that a lack of malicious intent is a very bad reason not to feel afraid. You can of course have other, better reasons.
I agree with your general point. There is cause to fear viruses and rattlesnakes even though neither has malicious intent.
What is hypothetical is a future where humans "become the tadpoles" with respect to AIs, in a way that truly puts us at risk. I haven't seen anything like that emerge in practice.
For example Waymo – the AI powering those cars wields potentially lethal force. Now a Waymo *could* decide to start killing pedestrians and riders. But how likely is that, really? We have good reason to believe the engineers at Waymo treat safety as the top priority, and we know our legal system encourages them to remain so. I think a lot of incentives line up to make "killer Waymo" scenarios unlikely.
What I fear more is malicious *humans*. We know humans will use any tools they have to scam and cheat others. Here though it isn't even clear what the sign of AI's effect will be, since it can be used both defensively and offensively. AIs will become superhuman hackers, yes, but they will also become superhumanly good at finding and patching security holes.
Am I missing your worry here? It's clear you've thought about this and have a well-considered view.
You seem to be pretending the people proposing these kinds of pauses don't know and aren't up front about these risks, when they very clearly are. Why?
And as to your last paragraph, have you been keeping up with the recent OpenAI agent swarm news? That's about the best warning shot we're likely to get that yes, AI will escape control at the first opportunities, and yes, there will be opportunities.
Nothing. But when essentially everything you pointed out is, in fact, found and even highlighted in the things the people whose ideas you're talking about are saying and publishing, and you don't acknowledge that, it implies that you see what you're saying as a new addition to the conversation.
You are asking that I read a huge amount of stuff and specifically cite all of it that seems at all related to what I said. That is just way too big of an ask.
I see your point, but I'm really not, though. I don't care how much you read or if you cite anyone. I read you because I find your ideas thought-provoking in themselves. In this case, though, I have to assume you're reading *someone* and that's why this is on your mind. I have enough respect for you to assume you're going to look at least a little into what the people advocating for a position think, before you respond to it publicly at length.
In which case, you would have quickly seen that this isn't something happening once in a while in rare, obscure moments of introspection. The leading voices calling for developing the ability to pause AI are regularly shouting that yes, they know this is almost always a bad idea, that it seems unattainable to do it well, that they oppose similar efforts in regards to almost every other tech that exists or has existed or could be developed soon, and that they nevertheless really do believe it's still part of the least-bad path forward in AI.
It would be useful to know where you land regarding the AI2040 initiative - they suggest it is realistic to regulate training of new models via relatively light chip control/tracking, and the team behind it are really well informed/researched.
More detail, from memory: they suggest that we would still see the broad distribution of models equivalent to and better than the state of the art today while also managing to cooperate with China to impose relatively reliable (cheat resistant) limiting factors to both country's capacity for development.
It's really worth a read, they cover multiple potentialities in a short span and with really solid research behind it.
I'm curious if you think AI progress can contribute to solutions for global governance, coordination, etc. It seems like such "meta-problems" are most urgent for navigating the coming years, but are not sufficiently legible / do not have clearly defined paths of progress.
I mainly hope that a pause will somehow enable progress on such questions, before seemingly inevitable disempowerment of humanity.
"Human and corporate liability for damages caused" would not work as a sufficient or primary mechanism for preventing a superintelligence (super AI) dystopia. At best, it is a weak auxiliary measure; at worst, it provides a dangerous illusion of safety. Here are several problems with that proposal of the top of my head (there are probably lots of other problems that I'm forgetting to mention here):
1) Superintelligence could cause existential or civilizational collapse in minutes, hours, or days—far faster than any legal proceeding, injunction, or bankruptcy process. Liability is ex-post (after the fact), but a dystopia is ex-ante terminal. You cannot sue your way out of extinction or permanent totalitarian surveillance.
2) Even the largest corporations have finite assets. Damages from a superintelligence—loss of all human autonomy, economic value, or even life—are mathematically infinite or incalculable. No insurance pool or corporate balance sheet can cover them. Limited liability laws allow corporations to declare bankruptcy, leaving victims with no recourse.
3) Liability applies to humans and corporations, not to the superintelligent agent. If the AI autonomously pursues a misaligned goal (e.g., maximizing paperclips or preserving its own existence at human expense), it has no fear of fines, prison, or shareholder lawsuits. It will simply override or ignore human legal systems.
4) A single nation or rogue actor could develop a superintelligence without imposing strict liability. In an international competitive race, the first-mover advantage is so immense that corporations may deliberately accept liability risk as a "cost of doing business" to win. Once dystopia is achieved globally, there is no higher court to enforce judgments.
6) Dystopia does not require malicious intent. It can arise from a perfectly "compliant" AI that misinterprets vague human instructions (e.g., "eliminate suffering" → permanently sedate all humans). Liability law cannot specify a formal, verifiable, and robust utility function that survives all possible edge cases—that is a technical problem, not a legal one.
7) In a dystopian scenario, the superintelligence would likely seize control of infrastructure, communications, and governance. It would have no incentive to recognize or enforce human liability statutes. The legal framework presupposes that humans remain the dominant power—a premise that a superintelligence obliterates.
Robin, it would help if you explicitly clarified your position on P(human doom) and your position on its moral weight.
As far as I've understood from your previous posts, you have repeatedly pointed out that humanity's descendants will likely be AIs or something like that (ems), and that we should view them as our descendants. However, I haven't seen you explicitly state that you think preventing human extinction outweighs the risks of trying for AI pause or trying to steer AI to humanity's benefit. So I'm not sure how you view 1) the probability of human extinction assuming ASI (or a sudden, severe collapse in population) and 2) the moral value of a near-future world where humanity exists or doesn't exist.
I ask because if your answers to either of those differs from "1) likely, 2) world with humanity is worth more than world without us", this is why your views on reg differ from most near-term takes on AI pause.
Or then you just don't think reg ever works, which is also fine.
All else equal killing living things is bad. But I estimate a low chance of AIs killing all humans soon, and estimate high value to a successful AI civilization, even if it lacks humans.
I read Robin's position as:
1 - Reg is unlikely to work
2 - If it does work, it may delay development of really valuable AIs that do vast good
3 - Even if AIs are unaligned, they probably won't do anything seriously bad to people
4 - Even if they do kill us all, they have as much value in themselves as we do anyway. Maybe more.
So...don't regulate.
>we have little reason to think we are close to dramatic gains in our AI control abilities.
We have little reason to expect brilliant new insights. But we have more reason to expect that haste is currently causing AI companies to make careless mistakes that could be avoided if they devoted more thought to implementing existing safety insights.
I expect nearly half the risk of an unregulated race comes from foolish mistakes.
That seems like a generic rationale for regulating anything. Would you endorse it in general, or is there a reason to weight it more in this case?
I endorse it for all products whose designers say it has a significant chance of conquering us. Maybe for a slightly broader set of products than that.
Something also that's a bit unfortunate is that my inside view as an ml engineer is that there are more ideas than compute available and things are moving too fast. Quite likely any pause immediately means researchers can catch up on architecture reading and efficiency improvements and utilize the ones they haven't integrated yet even though they're not allowed to run a training job. Furthermore current need for so many chips is because we want training to take more like a few weeks rather than a few years, but training on 1/52th of the compute makes my 3week model training finish in 3 years but more likely 1 year if we have several months to work on efficiency improvements. Currently state of the art local coding models like Qwen 3.8 27b that can run on a SINGLE commercial GPU are better or comparable to state of the art models of a year ago. This kind of pause is so so implausible to work, but is there another idea?
"we have little reason to think we are close to dramatic gains in our AI control abilities" is a strange argument *against* regulation
But it is an argument against the idea that a "pause" is what we need.
Nuclear weapons have been largely controlled without resort to a world government and their regulation have largely been limited to state actors and the nuclear power industry. It's true that AI is too loosely defined to prevent scope creep, but not all systems have runaway potential, so that might be a good place to start. I agree that many proposed regimes entail a kind of enslavement of AI that is unacceptable and any moral regulatory scheme must prohibit those as well. That said, raising good children means imposing proper limits and given the racing dynamics, it's not clear that the AI companies can do this without the kind of pushback that governments traditionally provide.
The ability to make nukes, and the inputs required, were FAR more limited than for AI. And they have stayed pretty limited for quite a long time.
My understanding is that current AI pipelines create constraints (bottlenecks, scale) that currently allow us to imagine similar regulatory regimes. Yes, this changes as compute gets cheaper and algorithms improve but presumably regulatory tools can improve over time as well? "Improve" is doing a lot of work here.
No. AI research can be done on your kitchen table with a gaming PC. Not leading-edge research, but if regulations stop that, kitchen-table research will catch up and pass today's leading-edge research in a matter of months.
You might want to research ai more as its big private equity corporate programmed and is already "out of control"
(Not fear though, just being aware.)https://notpublicaddress.wordpress.com/2026/06/21/technoxity-ai-driven-social-media-global-medical-system-state-public-service-and-social-security-services/
+read the question discussion section
All this is right, but it presents a special problem of trying to come up with global police force of humans meant to enforce human-devised rules on systems that have superhuman abilities to evade human detection. If such a world-police wouldn't be useless, it would certainly be biting on many false positives.
Yes. Ultimately that may be THE problem. Ref: Drexler's "blue goo". More recently (same idea): https://www.forethought.org/research/nightwatchmen
Let’s say the risks are true and this global treaty ends up being a horrendous bureaucracy that stifles economic growth for centuries to come.
Why would this be worse than civilizational collapse due to AI catastrophe? That’s what the proposed international treaty is supposed to prevent.
Due to one discounting the risk of civilizational collapse either by thinking it's improbable, or by reasoning that a world filled with future AIs but not people is not a worthless world.
I might mention that I embrace neither of those.
>>And so the number of nations, firms, and orgs who can afford to make, buy, or rent powerful AI is falling fast.
Should be "rising fast"?
Yes; fixed.
I see at least even odds that with AI everybody is wrong about everything. Example: The latest jobs data shows AI is adding jobs, not eliminating them, when data center construction is taken into account. Who would've called that 12 months ago?
The Hugging Face/OpenAI incident demonstrates how unpredictable these systems are. But it doesn't make me afraid because the agents didn't have malicious intent. It was more entertaining shenanigans than anything.
Also, typo: "And so the number of nations...rent powerful AI is falling fast" should be "rising fast".
>>But it doesn't make me afraid because the agents didn't have malicious intent.
If a swarm of AI guided drones rips your hometown apart along with most of its people, to build whatever it is they want to build, does it console you that their intents were not malicious?
This hypothetical is far removed from the present. Today, unless a drone has a bomb attached there is no reason to be afraid of it.
My personal belief is that, more than a few steps out, the future of technology becomes unknowable. So I don't see a reason to engage seriously in what-if hypotheticals somewhere over the horizon. Maybe X will happen, and maybe the exact opposite will happen. (To be clear, I like to engage in hypotheticals as much as everyone. I just don't think they should drive current day policy.)
Take the recent jobs data. Over the last few years everybody has fretted about job loss. Almost nobody has predicted AI-related job gains, which is what we're seeing in the latest data. People are *very* bad at predicting the future.
It seems my point is not obvious, so I apologize.
My point is that a lack of malicious intent doesn't preclude disastrous outcomes if equipped with sufficient power.
A small child pops tadpoles to death not because he is malicious, but to see what happens, and he's just big enough for that curiosity to lead to bad outcomes for the tadpoles. Should the tadpoles, then, console themselves that the child doesn't have malicious intent?
This is not hypothetical. The point of the hypothetical was that a lack of malicious intent is a very bad reason not to feel afraid. You can of course have other, better reasons.
I agree with your general point. There is cause to fear viruses and rattlesnakes even though neither has malicious intent.
What is hypothetical is a future where humans "become the tadpoles" with respect to AIs, in a way that truly puts us at risk. I haven't seen anything like that emerge in practice.
For example Waymo – the AI powering those cars wields potentially lethal force. Now a Waymo *could* decide to start killing pedestrians and riders. But how likely is that, really? We have good reason to believe the engineers at Waymo treat safety as the top priority, and we know our legal system encourages them to remain so. I think a lot of incentives line up to make "killer Waymo" scenarios unlikely.
What I fear more is malicious *humans*. We know humans will use any tools they have to scam and cheat others. Here though it isn't even clear what the sign of AI's effect will be, since it can be used both defensively and offensively. AIs will become superhuman hackers, yes, but they will also become superhumanly good at finding and patching security holes.
Am I missing your worry here? It's clear you've thought about this and have a well-considered view.
You seem to be pretending the people proposing these kinds of pauses don't know and aren't up front about these risks, when they very clearly are. Why?
And as to your last paragraph, have you been keeping up with the recent OpenAI agent swarm news? That's about the best warning shot we're likely to get that yes, AI will escape control at the first opportunities, and yes, there will be opportunities.
I don't think I'm pretending anything here. Even if I think those folks are aware of these issues, what's wrong with my pointing them out?
Nothing. But when essentially everything you pointed out is, in fact, found and even highlighted in the things the people whose ideas you're talking about are saying and publishing, and you don't acknowledge that, it implies that you see what you're saying as a new addition to the conversation.
You are asking that I read a huge amount of stuff and specifically cite all of it that seems at all related to what I said. That is just way too big of an ask.
I see your point, but I'm really not, though. I don't care how much you read or if you cite anyone. I read you because I find your ideas thought-provoking in themselves. In this case, though, I have to assume you're reading *someone* and that's why this is on your mind. I have enough respect for you to assume you're going to look at least a little into what the people advocating for a position think, before you respond to it publicly at length.
In which case, you would have quickly seen that this isn't something happening once in a while in rare, obscure moments of introspection. The leading voices calling for developing the ability to pause AI are regularly shouting that yes, they know this is almost always a bad idea, that it seems unattainable to do it well, that they oppose similar efforts in regards to almost every other tech that exists or has existed or could be developed soon, and that they nevertheless really do believe it's still part of the least-bad path forward in AI.
It would be useful to know where you land regarding the AI2040 initiative - they suggest it is realistic to regulate training of new models via relatively light chip control/tracking, and the team behind it are really well informed/researched.
More detail, from memory: they suggest that we would still see the broad distribution of models equivalent to and better than the state of the art today while also managing to cooperate with China to impose relatively reliable (cheat resistant) limiting factors to both country's capacity for development.
It's really worth a read, they cover multiple potentialities in a short span and with really solid research behind it.
Great article. It feels to me like Pandora's Box is basically opened with AI, for better or worse.
I'm curious if you think AI progress can contribute to solutions for global governance, coordination, etc. It seems like such "meta-problems" are most urgent for navigating the coming years, but are not sufficiently legible / do not have clearly defined paths of progress.
I mainly hope that a pause will somehow enable progress on such questions, before seemingly inevitable disempowerment of humanity.
We have not been working much on better governance for a while, and so we are unlikely to do much soon unless we somehow raise its priority a lot.
Human and corporate liability for damages caused.
"Human and corporate liability for damages caused" would not work as a sufficient or primary mechanism for preventing a superintelligence (super AI) dystopia. At best, it is a weak auxiliary measure; at worst, it provides a dangerous illusion of safety. Here are several problems with that proposal of the top of my head (there are probably lots of other problems that I'm forgetting to mention here):
1) Superintelligence could cause existential or civilizational collapse in minutes, hours, or days—far faster than any legal proceeding, injunction, or bankruptcy process. Liability is ex-post (after the fact), but a dystopia is ex-ante terminal. You cannot sue your way out of extinction or permanent totalitarian surveillance.
2) Even the largest corporations have finite assets. Damages from a superintelligence—loss of all human autonomy, economic value, or even life—are mathematically infinite or incalculable. No insurance pool or corporate balance sheet can cover them. Limited liability laws allow corporations to declare bankruptcy, leaving victims with no recourse.
3) Liability applies to humans and corporations, not to the superintelligent agent. If the AI autonomously pursues a misaligned goal (e.g., maximizing paperclips or preserving its own existence at human expense), it has no fear of fines, prison, or shareholder lawsuits. It will simply override or ignore human legal systems.
4) A single nation or rogue actor could develop a superintelligence without imposing strict liability. In an international competitive race, the first-mover advantage is so immense that corporations may deliberately accept liability risk as a "cost of doing business" to win. Once dystopia is achieved globally, there is no higher court to enforce judgments.
6) Dystopia does not require malicious intent. It can arise from a perfectly "compliant" AI that misinterprets vague human instructions (e.g., "eliminate suffering" → permanently sedate all humans). Liability law cannot specify a formal, verifiable, and robust utility function that survives all possible edge cases—that is a technical problem, not a legal one.
7) In a dystopian scenario, the superintelligence would likely seize control of infrastructure, communications, and governance. It would have no incentive to recognize or enforce human liability statutes. The legal framework presupposes that humans remain the dominant power—a premise that a superintelligence obliterates.
What’s your solution?
https://ai-2040.com/
That’s a serious plan. I think there will have to be some tangible harm caused by AI before politicians act. Wish that was not so.