I don’t think that explains his behavior well. And if it did, it would mean that he views us all as children, incapable of understanding the finer points, let alone actually contributing something.”

OB is broadcast to everyone?

Blunt statements of shocking conclusions are not that productive when they turn people off from considering the reasoning and general logic.

So you think that Eliezer avoids answering certain questions, because a little answer is a dangerous thing?

I don’t think that explains his behavior well. And if it did, it would mean that he views us all as children, incapable of understanding the finer points, let alone actually contributing something.

Tim Tyler wrote:

I am not sure about that. Eliezer’s proposed goal is a complicated extrapolation that neither I nor anyone else understands. Since the whole concept is pretty vague and amorphous, it seems rather difficult to say what it would actually do. Maybe it would kill people. However, you seemed to be claiming that it would be very likely to kill people.

I think that the possibility that it may kill people should be acknowledged. I think that in takeoff scenarios of a middling speed it is more likely to kill people; and that Eliezer and others have prematurely assigned all such scenarios a probability of zero.

My motive in pointing this out is not to say that it may kill people and this would be bad. My motive is more along the lines of prodding people out of thinking “if we can make friendly AI then we will be saved”.

My larger objective would be to point out that “we will be saved” is ill-defined; and that “saving humanity” will likely end up meaning something entailing the physical death of most humans, saving humans in a zoo with technological development forbidden, or something that we morally ought not to do.

The presentation of CEV is schizophrenic because of this. On one hand, it’s supposed to save human values by extrapolating from them. On the other hand, Eliezer thinks that values are arbitrary; and whenever he talks about CEV, he talks about saving the future for meatly humans – as if the purpose of guiding the AI’s development were not for its effects on the AI, but for its benefits for meat-humans. I don’t know if this is confusion on his part, or a deliberate schizophrenia cultivated to avoid scaring off donors. Repeated questioning by me has failed to produce any statement from him to the effect that he imagines the humans he wants to save ever being anything other than they are today.

Now, maybe you have a better understanding of Eliezer’s proposal than I do. However, the way the whole thing is formulated suggests you would have to be a superintelligent agent to actually make much sense of it. That makes it difficult for critics such as yourself to get much of a fix on the target.

Eliezer is not a superintelligent agent. So your statement necessarily implies that CEV is nonsense.

I would have a much better understanding of Eliezer’s proposal if he were willing to spend one-one-hundredth as much time answering simple questions about it, as he does writing about it.

But I also think I am done with it. I have wasted too much time already trying to bring about a review of someone else’s ideas, when that person isn’t even interested in having his ideas reviewed.

When people talk about the scientific method, they usually focus on the up-front part – making predictions and testing them. But another part of the scientific method is peer review. I can see how this would present problems for someone who imagines he has no peers. But “take it or leave it” is not operating within the scientific method.

You argued that “given an Eliezer-made AI, it is likely to kill people”.

I am not sure about that. Eliezer’s proposed goal is a complicated extrapolation that neither I nor anyone else understands. Since the whole concept is pretty vague and amorphous, it seems rather difficult to say what it would actually do. Maybe it would kill people. However, you seemed to be claiming that it would be very likely to kill people.

Now, maybe you have a better understanding of Eliezer’s proposal than I do. However, the way the whole thing is formulated suggests you would have to be a superintelligent agent to actually make much sense of it. That makes it difficult for critics such as yourself to get much of a fix on the target.

No matter how thoroughly we vet the ‘Friendliness’ of the AI, it is quite likely that the entity would eventually recommend some course of action that a person holding the plug would find unacceptable; said person would probably conclude the AI wasn’t really Friendly, and destroy it.

If someone managed to build an intelligence, and the intelligence concluded that six billion people needed to be killed within the next decade to create the least-worst outcome it could sense, do we really think people would accept this?

If the AI concluded that the past few thousand years of change in humanity’s moral systems were in error, do we really think people would reject their standards? Or would they insist the AI was in error because it rejected their standards and try to ‘fix’ it to reach the ‘correct’ results?

Don’t want the end to justify the means? Isn’t the solution simply to prioritise the means more highly? I mean, if you are concerned that the machine intelligence will murder people – and you don’t think any end justifies that – then ISTM that you can simply prioritise its conformance to legal constraints that prohibit killing.

Tim, you are one of the sharpest commentators here on OB, so it distresses me that after I have stated what I am trying to do, been misinterpreted, restated it, been misinterpreted in the same way again, and then restated it again, even you still can’t believe that I actually mean what I said.

I am NOT ARGUING FOR OR AGAINST drastic measures. Eliezer has never admitted that there is any potential for such drastic measures in his scenarios. I want people to realize that there is that potential, because it needs to enter into the conversation. Too many people are seduced by the word “friendly” in “Friendly AI” into thinking that it means the AI will be their friend. But the way Eliezer uses the term “friendly” isn’t even analogous to its true meaning.

Tyrell read me as saying a cliche which was almost the opposite of what I said. This is a known hazard in saying non-cliche things.

You’re right, I did misread you. My apologies.

Fair enought, esp. point (1)

Anyway, I have some faint feeling that (3) in fact describes “bussines as usual”, only with a new powerful agent. At least, we can hope that emerging AI will not destroy humanity abruptly, because it will need it for a long time… (ok, 72 hours minimum, that is).

But… I still cannot stop thinking that any sociopathic behaviour is not optimal, so as long as AI is really strong, it is not in its best interest to ‘misbehave’. Still, this depends on what it tries to optimize, which in the end is the purpose of this blog, right?

I’m not at present arguing over whether this hypothetical killing is good or bad. I just want people to realize that it is in the range of possible actions, and to own up to the fact that we are talking about a “the ends justifies the means” situation that would make Henry Kissinger flinch, rather than trusting that Papa AI will make everything better for everybody because he comes with a sticker that says “Friendly Inside!”

Don’t want the end to justify the means? Isn’t the solution simply to prioritise the means more highly? I mean, if you are concerned that the machine intelligence will murder people – and you don’t think any end justifies that – then ISTM that you can simply prioritise its conformance to legal constraints that prohibit killing.

ISTM that Eliezer’s plan amounts to trying to build a machine mind that voters in a democracy would approve of their government building – if they were consulted about the issue. Frankly, I doubt western voters would approve of a non-killing constraint – if it came to dealing swiftly with a proposed Islamic unkaafir (anti-infidel) superintelligence.