I agree. That’s a stark difference.

Find it hard to believe that that order of magnitude should make any difference. 1% is an unacceptable probability for an event as dangerous as that. So is 0.1%, come to think of it.

Besides, what margin of error would you apply here? Surely far more helpful (not to mention safer) to say ‘there’s an appreciable risk of X, and so studying X merits funds and man hours’.

This is not to say it won’t be interesting, but the dramatic lead-up makes this particular disagreement a little anti-climactic. Maybe there’s more to it. We’ll see.

My belief is that for AI to come about, there needs to be a fairly major scientific revolution. Once that revolution has come, the implementation of AI will be fairly straightforward (though still require a big effort). I think that revolution is on the horizon, I can see the storm clouds brewing, but it’s going to take 20 years or so. I don’t think that even heroic programming efforts can succeed until the new ideas arrive. People talk about incremental progress building up over many years, but in my view the work being done within current paradigms is achieving diminishing returns.

I agree with Eliezer that whole brain emulation is unlikely to succeed and with Robin that it is unlikely that a hand-coded version will suddenly awake and take over the world (depending on the time scale of “sudden”).

My personal fear is that fairly powerful but narrow AI (computer vision, speech recognition, etc) will be achieved in the near term and then used by governments to enslave their populations. The Singularity is a type of risk that has never before been faced but humanity has a chronic problem with totalitarianism. The disasters of the 21st century will probably just be the same as the 20th but with new technologies. I often consider abandoning AI research because of this fear.

Tim, yes, an eventual “singleton” mind is a different claim, and in my estimation much more likely, than that it arises suddenly and without warning.

Eliezer, I didn’t mean to imply you had originated the hard takeoff concept. But previous descriptions have been pretty hand-wavy compared to the detail usually worked out when making an argument in the economic growth literature. I want to know what you think is the best presentation and analysis of it, so that I can critique that.

If it happened, whether the route to the one would consist of an extended period of natural selection between corporate superintelligences – or if it would happen rapidly – e.g. as one government effectively took over the world also seems not to be known. We can see large technological shifts looming – in the form of machine intelligence and nanotechnology. They may well provide enough of a power-imbalance to result in an opportunity for such a shift.

These seem like rather separate issues to me. Will we wind up with one big superintelligence? is one issue… and if that happens, will it be a direct descendant of the first superintelligence? is another one. IMO, it is best to consider these issues quite separately – they represent quite different problems.

Corporate competition today seems relatively benign today – but competition between future superintelligent agents might not be.

The possibility of superintelligent agents competing with one another on the planet for their respective futures does seem like a rather worrying one to me.

On the other hand, at least natural selection might help save us from the fate of a sexually-selected species which pursues its own evolution down a blind alley. On this front, it’s hard to even know what we should want – let alone what we will get.