88 Comments

"So you think that Eliezer avoids answering certain questions, because a little answer is a dangerous thing?

I don't think that explains his behavior well. And if it did, it would mean that he views us all as children, incapable of understanding the finer points, let alone actually contributing something."

OB is broadcast to everyone?

Expand full comment

Spambot wrote:

Blunt statements of shocking conclusions are not that productive when they turn people off from considering the reasoning and general logic.So you think that Eliezer avoids answering certain questions, because a little answer is a dangerous thing?

I don't think that explains his behavior well. And if it did, it would mean that he views us all as children, incapable of understanding the finer points, let alone actually contributing something.

Tim Tyler wrote:

I am not sure about that. Eliezer's proposed goal is a complicated extrapolation that neither I nor anyone else understands. Since the whole concept is pretty vague and amorphous, it seems rather difficult to say what it would actually do. Maybe it would kill people. However, you seemed to be claiming that it would be very likely to kill people.I think that the possibility that it may kill people should be acknowledged. I think that in takeoff scenarios of a middling speed it is more likely to kill people; and that Eliezer and others have prematurely assigned all such scenarios a probability of zero.

My motive in pointing this out is not to say that it may kill people and this would be bad. My motive is more along the lines of prodding people out of thinking "if we can make friendly AI then we will be saved".

My larger objective would be to point out that "we will be saved" is ill-defined; and that "saving humanity" will likely end up meaning something entailing the physical death of most humans, saving humans in a zoo with technological development forbidden, or something that we morally ought not to do.

The presentation of CEV is schizophrenic because of this. On one hand, it's supposed to save human values by extrapolating from them. On the other hand, Eliezer thinks that values are arbitrary; and whenever he talks about CEV, he talks about saving the future for meatly humans - as if the purpose of guiding the AI's development were not for its effects on the AI, but for its benefits for meat-humans. I don't know if this is confusion on his part, or a deliberate schizophrenia cultivated to avoid scaring off donors. Repeated questioning by me has failed to produce any statement from him to the effect that he imagines the humans he wants to save ever being anything other than they are today.

Now, maybe you have a better understanding of Eliezer's proposal than I do. However, the way the whole thing is formulated suggests you would have to be a superintelligent agent to actually make much sense of it. That makes it difficult for critics such as yourself to get much of a fix on the target.Eliezer is not a superintelligent agent. So your statement necessarily implies that CEV is nonsense.

I would have a much better understanding of Eliezer's proposal if he were willing to spend one-one-hundredth as much time answering simple questions about it, as he does writing about it.

But I also think I am done with it. I have wasted too much time already trying to bring about a review of someone else's ideas, when that person isn't even interested in having his ideas reviewed.

When people talk about the scientific method, they usually focus on the up-front part - making predictions and testing them. But another part of the scientific method is peer review. I can see how this would present problems for someone who imagines he has no peers. But "take it or leave it" is not operating within the scientific method.

Expand full comment

"I don't think I can say it any more clearly: I'm not at present arguing over whether this hypothetical killing is good or bad."I knew that, and agree that a CEV AI might kill quite a lot of people (in general I agree with Caledonian's comment above), perhaps depending quite sensitively on the aggregation and extrapolation rules. I explained why I think it's not very valuable to wrangle about the specifics, and why I was uninterested in Eliezer doing so and didn't mind that he hasn't spent a huge amount of time shocking people about it. Blunt statements of shocking conclusions are not that productive when they turn people off from considering the reasoning and general logic.

Expand full comment

Phil, thanks! Maybe it's just that our views are usually pretty similar...

You argued that "given an Eliezer-made AI, it is likely to kill people".

I am not sure about that. Eliezer's proposed goal is a complicated extrapolation that neither I nor anyone else understands. Since the whole concept is pretty vague and amorphous, it seems rather difficult to say what it would actually do. Maybe it would kill people. However, you seemed to be claiming that it would be very likely to kill people.

Now, maybe you have a better understanding of Eliezer's proposal than I do. However, the way the whole thing is formulated suggests you would have to be a superintelligent agent to actually make much sense of it. That makes it difficult for critics such as yourself to get much of a fix on the target.

Expand full comment

Mr. Goetz is quite right.

No matter how thoroughly we vet the 'Friendliness' of the AI, it is quite likely that the entity would eventually recommend some course of action that a person holding the plug would find unacceptable; said person would probably conclude the AI wasn't really Friendly, and destroy it.

If someone managed to build an intelligence, and the intelligence concluded that six billion people needed to be killed within the next decade to create the least-worst outcome it could sense, do we really think people would accept this?

If the AI concluded that the past few thousand years of change in humanity's moral systems were in error, do we really think people would reject their standards? Or would they insist the AI was in error because it rejected their standards and try to 'fix' it to reach the 'correct' results?

Expand full comment

Don't want the end to justify the means? Isn't the solution simply to prioritise the means more highly? I mean, if you are concerned that the machine intelligence will murder people - and you don't think any end justifies that - then ISTM that you can simply prioritise its conformance to legal constraints that prohibit killing.Tim, you are one of the sharpest commentators here on OB, so it distresses me that after I have stated what I am trying to do, been misinterpreted, restated it, been misinterpreted in the same way again, and then restated it again, even you still can't believe that I actually mean what I said.

I am NOT ARGUING FOR OR AGAINST drastic measures. Eliezer has never admitted that there is any potential for such drastic measures in his scenarios. I want people to realize that there is that potential, because it needs to enter into the conversation. Too many people are seduced by the word "friendly" in "Friendly AI" into thinking that it means the AI will be their friend. But the way Eliezer uses the term "friendly" isn't even analogous to its true meaning.

Expand full comment

Tyrell read me as saying a cliche which was almost the opposite of what I said. This is a known hazard in saying non-cliche things.

You're right, I did misread you. My apologies.

Expand full comment

Eliezer:

Fair enought, esp. point (1) :)

Anyway, I have some faint feeling that (3) in fact describes "bussines as usual", only with a new powerful agent. At least, we can hope that emerging AI will not destroy humanity abruptly, because it will need it for a long time... (ok, 72 hours minimum, that is).

But... I still cannot stop thinking that any sociopathic behaviour is not optimal, so as long as AI is really strong, it is not in its best interest to 'misbehave'. Still, this depends on what it tries to optimize, which in the end is the purpose of this blog, right?

Expand full comment

I'm not at present arguing over whether this hypothetical killing is good or bad. I just want people to realize that it is in the range of possible actions, and to own up to the fact that we are talking about a "the ends justifies the means" situation that would make Henry Kissinger flinch, rather than trusting that Papa AI will make everything better for everybody because he comes with a sticker that says "Friendly Inside!"

Don't want the end to justify the means? Isn't the solution simply to prioritise the means more highly? I mean, if you are concerned that the machine intelligence will murder people - and you don't think any end justifies that - then ISTM that you can simply prioritise its conformance to legal constraints that prohibit killing.

ISTM that Eliezer's plan amounts to trying to build a machine mind that voters in a democracy would approve of their government building - if they were consulted about the issue. Frankly, I doubt western voters would approve of a non-killing constraint - if it came to dealing swiftly with a proposed Islamic unkaafir (anti-infidel) superintelligence.

Expand full comment

billswift: "The real risks are either a spurious optimization gets accidentally programmed into it (paperclip maximizer type situation) or there is a serious attempt on the part of society to destroy it and it has a survival "instinct". I think the latter is the bigger risk."

denis bider: "On the contrary, it seems to me that the first part is the vastly bigger risk. Attempting not to accidentally program a spurious optimization is like attempting to write a complex piece of software with no bugs, and have it run without error on the first time. You don't get a second chance, and if you've so much as entered a comma where a semicolon should have been, it very well could be the end of things."

I actually meant "goal" where I typed optimization, I've been reading Eliezer too much.

I don't expect low-level programming bugs to be too much of a problem, it's the design errors I'm mostly worried about. But I still expect the latter to be a bigger risk because of human irrationality. I think too many on this, and most of the other techie type blogs I follow, are too much out of touch with "normal" people to realize just how irrationally many of them are likely to act.

Expand full comment

(Phil) The important question is whether Eliezer respects Robin enough to say that, given that Robin thinks there is a high chance of a slow takeoff, Eliezer will create some contingency plan in case of a slow takeoff.

Eliezer's plan, if followed, may lead to disastrous consequences if the takeoff is not hard.----Good point. I'd be intrigued to see a post on that. I imagine there are all sorts of complications, subtleties and biasses involved in taking precautions against things that (most of) you do not believe to happen. Even an analysis by Eliezer of how is strategy would be different if he held Robin's assumptions rather than his own would be interesting.

Expand full comment

It's something of a fool's game for Eliezer to defend a vast series of propositions of the form, 'well, if CEV outputs strategy X on the basis of belief Y and value Z (adopted by our extrapolated selves after reflection), there must be a good reason for Y, and good reasons for Z if we were to ponder the matter' for different values of X, Y, and Z. I can put in all sorts of taboo beliefs for 'Y' from the list below to create 'gotchas,' but it won't really be productive.I think you missed the point of my answer to you; which is odd, since you quoted it at the start of your reply. I don't think I can say it any more clearly: I'm not at present arguing over whether this hypothetical killing is good or bad. I just want people to realize that it is in the range of possible actions, and to own up to the fact that we are talking about a "the ends justifies the means" situation that would make Henry Kissinger flinch, rather than trusting that Papa AI will make everything better for everybody because he comes with a sticker that says "Friendly Inside!"

Realizing that this is the domain we are working in is a prerequisite for any meaningful critique.

Expand full comment

"Second, I'm not arguing over whether this hypothetical killing is good or bad. I'm just trying to get it out into the open."

What if the AI concludes that Jesus Christ was not a supernatural being capable of granting infinite bliss to its followers, and convinces people that Christianity is false? A Christian fundamentalist today would object to an AI that he expected to do that, but could endorse an AI that would seek out the truth impartially and act as he would with that knowledge, expecting that this would result in a Christian AI. For him to really fear the results of such truth-seeking, he would have to think that his beliefs might be false with some substantial probability, and in that case he wouldn't want to believe them. Predictions about consequences of a CEV AI depend crucially on a lot of facts about what we would want if we knew more and were smarter, physics, etc.

It's something of a fool's game for Eliezer to defend a vast series of propositions of the form, 'well, if CEV outputs strategy X on the basis of belief Y and value Z (adopted by our extrapolated selves after reflection), there must be a good reason for Y, and good reasons for Z if we were to ponder the matter' for different values of X, Y, and Z. I can put in all sorts of taboo beliefs for 'Y' from the list below to create 'gotchas,' but it won't really be productive.

http://richarddawkins.net/a...

Expand full comment

"the groundless claim that it will be so much more clever than us that it will resolve all threats and conflicts painlessly. This attitude is religious, not rational."

I think Phil is making sense.

If friendliness is a relevant pursuit, it *must* be because there is a threat of powerful unfriendly AI's. These unfriendly AI's would have the power moreso than any other optimizing force in the near universe to do harm to humanity. Therefore, it is logical to believe that preventing unfriendly AI's would become one of the top missions of a friendly AI. Therefore, the only question is what it would take for the friendly AI to accomplish its goal particularly if killing people leads to a much higher probability of success (e.g. the unfriendly AI is being developed in Iran who were smart enough to put up good enough firewalls against the pig capitalist AI across the pond).

And possessing the friendliest AI doesn't guarantee you to have the smartest or most powerful AI, nor does it mean that the AI can diffuse all scenarios without bloodshed even waving the magic sceptre of perfect analytical bayesian statistics +3.

Expand full comment

He seems to be saying that he thinks that if humanity was much smarter and extrapolated, we would be willing to risk our lives to ensure a bright future, but that either his preferences are idiosyncratic or that he disagrees with what he would want if fully informed.I think you're saying this because<ul><li>I've said there's reason to believe that E's AI will kill people.</li><li>I seem to disapprove of this.</li></ul>

First, you're assuming that I actually agree with E's many assumptions that go into that AI's decisions. I don't.

Second, I'm not arguing over whether this hypothetical killing is good or bad. I'm just trying to get it out into the open. Perhaps it can be justified. But at present most of the OB community is giving Eliezer a pass on having to defend, or even spell out, the consequences of his plans. Eliezer's pronouncements on the matter amount to, "My AI will be sweet and kind to everybody so don't worry about it." Any suggestion that this AI would have to make hard decisions are countered, by Eliezer and others, with the groundless claim that it will be so much more clever than us that it will resolve all threats and conflicts painlessly. This attitude is religious, not rational.

Expand full comment

"If no one besides Goetz thinks that Goetz is making sense, I won't respond to him further."

He seems to be saying that he thinks that if humanity was much smarter and extrapolated, we would be willing to risk our lives to ensure a bright future, but that either his preferences are idiosyncratic or that he disagrees with what he would want if fully informed. Or he may be objecting to a hypothesized procedure of aggregation (different aggregation rules will lead to different results) or extrapolation, expecting that you would favor an aggregation rule that favors disinterested consequentialism.

Expand full comment