The big difference between AI and humans is speed. Maybe my distant descendants will turn into some kind of world-destroying doomsday cult, but at least I won’t be around to experience it—whereas with AI there’s no such guarantee. Additionally, we have some degree of understanding of human psychology and social systems, and could make interventions like your futarchy idea. No such understanding exists for AI.
We are making AIs in human psych image, by training them on our text. Most of our understanding of institutions doesn't depend much on human details; game theory agents do fine.
I'm unhappy with Bryan Caplan's and Richard Hanania's take ("nothing to worry about - just an improved steam engine"). I respect them both and think they should see the danger more clearly.
My main worry (still) is RSI leading to an AI that can dominate competing AIs, with goals that make it see humanity as an obstacle. The concern is the one AI, not the median AI. (Value drift is a concern *too*, but a less immediate one. Worst-case human value drift seems to lead to a world of Mormons or Amish - not so bad.)
Yet attempts to ban/regulate seem likely to make things worse. That drives research underground, to people less savory, less responsible, more willing to accept risk.
If super-intelligent AI is going to happen, it’s already too late to stop it – there are thousands of people all over the world, and millions of gaming PCs, able to do AI research (not leading edge but not so far behind that progress can’t be made). Some will do it.
I don't think we're doomed, but I do see serious risk.
I think the best we can do is proceed with caution - make sure the people who do it first (and if they do a good job, will have super-AI capable of controlling others that come later) are careful, responsible, not under pressure to cut corners. Probably that means letting the big AI labs do it – at least if they’re doing it under public, investor, and government scrutiny, and are basically law-abiding type people, they’re less likely to take crazy risks.
Underground, secret military, or criminal projects seem less likely to be careful.
I agree that humans are as great or a greater risk to ourselves than AI. I would even add that AI can in theory be part of the solution to this problem.
Where I differ is in the extremely narrow focus on cultural drift. I think this is a very specific subset of potential risks which can include unintended consequences, out of control arms races, mistakes, and coordination failures.
Moloch can get us with or without AI.
AI may help us escape some Moloch traps—or deepen and quicken them.
I also have to point out that the thing that would cause AIs to have adaptive values and avoid cultural drift—Darwinian competition—would be most likely to lead to human extinction, as humans became collateral damage in struggles between AI swarms over resources. Consider how the colonization of Amerindian lands in the New World was driven by European settlers competing with other European settlers for land, and European empires competing with other European empires for global advantage.
The big difference between AI and humans is speed. Maybe my distant descendants will turn into some kind of world-destroying doomsday cult, but at least I won’t be around to experience it—whereas with AI there’s no such guarantee. Additionally, we have some degree of understanding of human psychology and social systems, and could make interventions like your futarchy idea. No such understanding exists for AI.
Our understanding of psychology and institutions are also quite applicable to AI.
That’s a claim that needs to be justified, given the massive apparent difference between AI agents and humans.
We are making AIs in human psych image, by training them on our text. Most of our understanding of institutions doesn't depend much on human details; game theory agents do fine.
I'm unhappy with Bryan Caplan's and Richard Hanania's take ("nothing to worry about - just an improved steam engine"). I respect them both and think they should see the danger more clearly.
My main worry (still) is RSI leading to an AI that can dominate competing AIs, with goals that make it see humanity as an obstacle. The concern is the one AI, not the median AI. (Value drift is a concern *too*, but a less immediate one. Worst-case human value drift seems to lead to a world of Mormons or Amish - not so bad.)
Yet attempts to ban/regulate seem likely to make things worse. That drives research underground, to people less savory, less responsible, more willing to accept risk.
If super-intelligent AI is going to happen, it’s already too late to stop it – there are thousands of people all over the world, and millions of gaming PCs, able to do AI research (not leading edge but not so far behind that progress can’t be made). Some will do it.
I don't think we're doomed, but I do see serious risk.
I think the best we can do is proceed with caution - make sure the people who do it first (and if they do a good job, will have super-AI capable of controlling others that come later) are careful, responsible, not under pressure to cut corners. Probably that means letting the big AI labs do it – at least if they’re doing it under public, investor, and government scrutiny, and are basically law-abiding type people, they’re less likely to take crazy risks.
Underground, secret military, or criminal projects seem less likely to be careful.
If one AI has most power, it is the median.
I agree that humans are as great or a greater risk to ourselves than AI. I would even add that AI can in theory be part of the solution to this problem.
Where I differ is in the extremely narrow focus on cultural drift. I think this is a very specific subset of potential risks which can include unintended consequences, out of control arms races, mistakes, and coordination failures.
Moloch can get us with or without AI.
AI may help us escape some Moloch traps—or deepen and quicken them.
I also have to point out that the thing that would cause AIs to have adaptive values and avoid cultural drift—Darwinian competition—would be most likely to lead to human extinction, as humans became collateral damage in struggles between AI swarms over resources. Consider how the colonization of Amerindian lands in the New World was driven by European settlers competing with other European settlers for land, and European empires competing with other European empires for global advantage.