He is right, if this works out maybe putting your money there would be a bad idea IF THE PROJECT worked, for it would be so dramatic a change that maybe being distant from it was more palatable. But this is also not the question, the question is “What could make you believe it to work?” My own opinion follows.

First, i would require the enterprise to NOT frame it’s aims in recognizable terms. That means, i would not believe anyone proposing to make a “new computer” or a “new telescope” or even a “new industry” or to shorten any NEW anything. Any advancement that really did merit the “über” in the title would be so new that it would be ridiculous to explain it in terms of previous data.

Also, i guess i would expect them to have huge communication problems. Anyone who had a reasonable plan to get there would be so hopelessly beyond current trends as to make it difficult to understand, so to say. (This is actually a side-effect from my first proposition).

I would require the plan to incorporate present, common-sense, familiar, reliable ideas in unfamiliar and unreliable and unpredictable ways. That is to say, i would expect the plan to begin with present techniques, without dependence upon big-tech or big-science, so that the elements could be expected to mix and match in creative ways without big costs at each interaction.

Finally, i think the general realm where the advancements would be expected to apply must be very mundane and “day-to-day”, that is, i would expect the proposed “final stage” to be spoken of not in terms of some magical-sounding arcane “super-power”, but instead in terms of modifications in the experience of concrete human beings. I think the enterprise would have to be aiming not at “being the next technological wave”, but in “changing our very way of life”, if you can see the difference.

But obviously, the very issue of “why don’t we simply do that” is noise in the chain. The meta-advancement does not happen just because it is difficult to get rid of the small stupid problems. It is difficult to diminish the inertia. If they could tell me how they expect to do that, i would not only be already putting my money there, i would be applying for a job!

But there is a non-objective way of arbitrating. It’s called history. Get out of your ivory tower, people: if you believe in the battle for friendliness, it is being fought on the street right outside right now.

“Humanity has a collection of drives and desires, as does each one of us, but these desires conflict, and there is no objective way of arbitrating.”

Oh this is such an important idea, Roko, and one most memorably and beautifully stated by its father, Sir Isaiah Berlin, in his book Liberty. He called it “value pluralism.” I highly recommend you look at Liberty – one of the greatest and most gorgeously written books of philosophy in the 20th century – and also his further thoughts on the idea in his Crooked Timber of Humanity.

Indeed.

What, on might ask, does it mean to make the collection of all human minds “friendly”? Given that I am now a convert to moral anti-realism, I am not entirely sure that this question has a well-defined answer.

And I think that this is basically the core problem: there is no clean definition of what the “right” thing to do is, so “friendly X” doesn’t actually make much sense in general. Humanity has a collection of drives and desires, as does each one of us, but these desires conflict, and there is no objective way of arbitrating.

(OT) A big topic of this blog is taming AI to be friendly. The assumption is that thinking philosophically about how that might be possible is the most promising path for achieving FAI. I think that it would be just as promising, and far more reliably productive, to try to tame-to-friendliness the other advancements which are seen as analogous to AI – that is, human minds and society, agriculture, and industrial capitalism. All three are showing increasing signs of leading towards decidedly unfriendly outcomes, especially (but not exclusively) when considered in light of their predecessor’s value systems. If the AI outcome is path-dependent, I find it stretches plausibility that the friendliness or otherwise of these larger systems will not play a corresponding role in determining that path.

What I’m saying is that extrapolating a double-exponential, which may be purely an anthropic artifact, into the future is very shaky ground without some strong supporting evidence.

Allow me to speculate: Robin is about to post that, because of FHI’s brain emulation roadmap and the comments such as

“I want to see the working prototype or proof-of-concept and testimony from outside experts on the tech”

“So the venture capitalist’s criteria should include a credible road map of useful intermediate stages.”

and, of course, the complete “shot in the dark” that AGI represents, that no-one is going to invest in AGI when brain emulation is there as an alternative.

And I quite agree; the only thing that could upset this prediction is if brain emulation turns out to be moderately harder than we expect – and according to the Road Map there are ways that this could happen – whilst someone comes up with a great innovation in AI, for example a really good way of representing knowledge, or theoretical progress on self-improving AI. Given the large number of research years spent on AGI, this is only moderately likely by, say, 2040.

So somebody needs to start thinking very hard about the social and ethical consequences of an upload singularity. How does this change things?