Pause-AI will never fly. AI research is too distributed, too accessible, and too potentially valuable. No politician will stand in front of that freight train. Moreover you can't learn while you're paused. Technology is unpredictable and we learn by running the experiment.
A more practical question is highlighted by the US government's pause on the Fable rollout. I.e., how should we release superhuman models without creating undue short-term risk.
This is the second recent post in which you seem to have changed your mind re AGI and ASI - formerly you thought these were decades if not centuries away. Care to elaborate on your change of view?
I agree - AI will be our descendants; I'm with Moravec on that and am personally more concerned with the survival of intelligence and consciousness in the universe than of my genes. Those are not necessarily mutually exclusive, opinions differ. But still, we don't let children drive cars - or do lots of other things they'd like to do - until they're mature enough to do so with reasonable safety to themselves and others. Maybe AIs also will need some time for that.
I'd really hate to see a universe of dead-end paperclip maximisers, snuffing out all competition.
I'm curious: have you changed the timescale you expect at all? E.g. does next century still seem fairly likely for how long fully general superseding of human intellect (e.g. w/ TED-AI, HL-AI or drop-in remote worker operationalizations) will take?
A lot of species have gone extinct, outcompeted by closely related species occupying the same ecological niche. Of course, lots of species go extinct without that either.
I'm curious what big changes blocked indefinitely Hanson is referring to, re: "They’ve typically just blocked change indefinitely." Pausing and slowing big changes seems rational to me, more rational than the accelerate and break things strategy Hanson is warning against. But maybe that's because I don't know what examples of "blocked indefinitely" he is referring to.
Ah yes, the "potential" for most technologies is always vastly larger, right? We could have made and disseminated trains, cars, planes, antibiotics, computers, etc, all much faster. But are you agreeing that nuclear was not really blocked, just not used to its potential?
Given how much better wind and solar seem to work, can we agree that looking at trends from a long-term perspective, nuclear energy was always going to be at best an intermediate step between fossil fuels and renewables?
Nuclear was effectively blocked due to the way cost-effectiveness was legally prevented by definition (requiring best reasonably possible safety practices when you can always eke out some incremental extra safety precautions or equipment or whatever instead of actually saving on costs).
So far as I'm aware, there's no "control group" to see how cost-effective nuclear power could be relative to wind and solar, since all countries over-regulate their nuclear power more than other energy infrastructure.
If you take a really long term perspective, then sure solar is going to dominate all other energy sources (including wind, actually not sure about geothermal) because the total energy hitting the earth from sunlight is so large. But that's quite a few doublings of gdp past the present day.
Lots of changes are fought by those who see excessive risk in them.
When the opponents lose, the change happens, and historically the change is usually for the better.
But if the opponents are correct, they usually win, and the change never happens. So we can't just say "change is usually good, don't fight it". We should oppose changes we think are net bad, and support those we think are net good, and accept that if we lose we were probably wrong.
That doesn't change your conclusion - probably the doomers are wrong. Probably.
--
BTW, *how* would you "set up insurance re job loss"? Do you literally mean insurance? Who would write such a policy? Who could afford to pay out?
The evidence you offer for the cost of blocking comes from regional cases, where other regions continue and provide an imperfect comparison. A global pause would have no outside adopter during the pause to show what waiting costs.
You say you need concrete reasons to treat AI as unusually dangerous. What kind of predeployment evidence could outweigh the regional record when the policy case is global?
Pause-AI will never fly. AI research is too distributed, too accessible, and too potentially valuable. No politician will stand in front of that freight train. Moreover you can't learn while you're paused. Technology is unpredictable and we learn by running the experiment.
A more practical question is highlighted by the US government's pause on the Fable rollout. I.e., how should we release superhuman models without creating undue short-term risk.
Typo: "But Idon’t see".
This is the second recent post in which you seem to have changed your mind re AGI and ASI - formerly you thought these were decades if not centuries away. Care to elaborate on your change of view?
I agree - AI will be our descendants; I'm with Moravec on that and am personally more concerned with the survival of intelligence and consciousness in the universe than of my genes. Those are not necessarily mutually exclusive, opinions differ. But still, we don't let children drive cars - or do lots of other things they'd like to do - until they're mature enough to do so with reasonable safety to themselves and others. Maybe AIs also will need some time for that.
I'd really hate to see a universe of dead-end paperclip maximisers, snuffing out all competition.
I said nothing about timescales here.
I'm curious: have you changed the timescale you expect at all? E.g. does next century still seem fairly likely for how long fully general superseding of human intellect (e.g. w/ TED-AI, HL-AI or drop-in remote worker operationalizations) will take?
> While yes there’s a risk these descendants killing their ancestors, that’s hardly something to expect.
Why not? I'd say that it's very much something to expect. The history of evolution is rife with such cases.
Expecting an event is assigning a >50% chance. "Rife with" doesn't meet that standard.
A lot of species have gone extinct, outcompeted by closely related species occupying the same ecological niche. Of course, lots of species go extinct without that either.
I'm curious what big changes blocked indefinitely Hanson is referring to, re: "They’ve typically just blocked change indefinitely." Pausing and slowing big changes seems rational to me, more rational than the accelerate and break things strategy Hanson is warning against. But maybe that's because I don't know what examples of "blocked indefinitely" he is referring to.
Nuclear energy is the classic example.
"Blocked indefinitely"? My Perplexity says (I have not checked the accuracy, but you'll get the point):
United States: 94 operating reactors.
China: 63 operating reactors.
France: 57 operating reactors.
Russia: 34 operating reactors.
South Korea: 26 operating reactors.
The potential was vastly larger
Ah yes, the "potential" for most technologies is always vastly larger, right? We could have made and disseminated trains, cars, planes, antibiotics, computers, etc, all much faster. But are you agreeing that nuclear was not really blocked, just not used to its potential?
Given how much better wind and solar seem to work, can we agree that looking at trends from a long-term perspective, nuclear energy was always going to be at best an intermediate step between fossil fuels and renewables?
No I don't agree.
Nuclear was effectively blocked due to the way cost-effectiveness was legally prevented by definition (requiring best reasonably possible safety practices when you can always eke out some incremental extra safety precautions or equipment or whatever instead of actually saving on costs).
So far as I'm aware, there's no "control group" to see how cost-effective nuclear power could be relative to wind and solar, since all countries over-regulate their nuclear power more than other energy infrastructure.
If you take a really long term perspective, then sure solar is going to dominate all other energy sources (including wind, actually not sure about geothermal) because the total energy hitting the earth from sunlight is so large. But that's quite a few doublings of gdp past the present day.
Lots of changes are fought by those who see excessive risk in them.
When the opponents lose, the change happens, and historically the change is usually for the better.
But if the opponents are correct, they usually win, and the change never happens. So we can't just say "change is usually good, don't fight it". We should oppose changes we think are net bad, and support those we think are net good, and accept that if we lose we were probably wrong.
That doesn't change your conclusion - probably the doomers are wrong. Probably.
--
BTW, *how* would you "set up insurance re job loss"? Do you literally mean insurance? Who would write such a policy? Who could afford to pay out?
The evidence you offer for the cost of blocking comes from regional cases, where other regions continue and provide an imperfect comparison. A global pause would have no outside adopter during the pause to show what waiting costs.
You say you need concrete reasons to treat AI as unusually dangerous. What kind of predeployment evidence could outweigh the regional record when the policy case is global?