AI Go Foom

It seems to me that it is up to [Eliezer] to show us how his analysis, using his abstractions, convinces him that, more likely than it might otherwise seem, hand-coded AI will come soon and in the form of a single suddenly super-powerful AI.

As this didn’t prod a response, I guess it is up to me to summarize Eliezer’s argument as best I can, so I can then respond.  Here goes:

A machine intelligence can directly rewrite its entire source code, and redesign its entire physical hardware.  While human brains can in principle modify themselves arbitrarily, in practice our limited understanding of ourselves means we mainly only change ourselves by thinking new thoughts.   All else equal this means that machine brains have an advantage in improving themselves. 

A mind without arbitrary capacity limits, that focuses on improving itself, can probably do so indefinitely.  The growth rate of its "intelligence" may be slow when it is dumb, but gets faster as it gets smarter.  This growth rate also depends on how many parts of itself it can usefully change.  So all else equal, the growth rate of a machine intelligence must be greater than the growth rate of a human brain. 

No matter what its initial disadvantage, a system with a faster growth rate eventually wins.  So if the growth rate advantage is large enough then yes a single computer could well go in a few days from less than human intelligence to so smart it could take over the world.  QED.

So Eliezer, is this close enough to be worth my response?  If not, could you suggest something closer?

GD Star Rating
loading...
Tagged as:
Trackback URL:
  • http://don.geddis.org/ Don Geddis

    “A few days” is probably not the critical part. It could take six months or a year, to go from below-human to take-over-the-world, and that would probably still be more than fast enough that humanity could not adapt to control it successfully.

  • http://yudkowsky.net/ Eliezer Yudkowsky

    Well, the format of my thesis is something like,

    “When you break down the history of optimization into things like optimization resource, optimization efficiency, and search neighborhood, and come up with any reasonable set of curves fit to the observed history of optimization so far including the very few points where object-level innovations have increased optimization efficiency, and then you try to fit the same curves to an AI that is putting a large part of its present idea-production flow into direct feedback to increase optimization efficiency (unlike human minds or any other process witnessed heretofore), then you get a curve which is either flat (below a certain threshold) or FOOM (above that threshold).”

    If that doesn’t make any sense, it’s cuz I was rushed.

    Roughly… suppose you have a flat linear line, and this is what happens when you have a laborer pushing on a wheelbarrow at constant speed. Now suppose that the wheelbarrow’s speed is proportional to the position to which it has been pushed so far. Folding a linear graph in on itself will produce an exponential graph. What we’re doing is, roughly, taking the graph of humans being pushed on by evolution, and science being pushed on by humans, and folding that graph in on itself. The justification for viewing things this way has to do with asking questions like “Why did Eurisko run out of steam?”, and “Why can’t you keep running an optimizing compiler on its own source code to get something faster and faster?”, and considering the degree to which meta-level functions can get encapsulated or improved by object-level pressures, which determines the strength of the connections in the positive feedback loop.

    I was rushed, so don’t blame me if that doesn’t make sense either.

    Consider that as my justification for trying to answer the question in a post, rather than a comment.

    It seems to me that we are viewing this problem from extremely different angles, which makes it more obvious to each of us that the other is just plain wrong, than we trust in the others’ rationality; and this is the result of the persistent disagreement. It also seems to me that you expect that you know what I will say next, and are wrong about this, whereas I don’t feel like I know what you will say next. It’s that sort of thing that makes me reluctant to directly jump to your point in opinion-space having assumed that you already took mine fully into account.

  • Tim Tyler

    How about the concepts of “first mover advantage” and “winner-takes-all ecosystem”.

    Microsoft had a utterly crappy OS, literally dreadful – and yet they still have a virtual monopoly on the market 25 years later – despite anti-trust measures by the government. The main reason seems to be that they got there first.

  • Tim Tyler

    Still, the end may be in sight: http://google.com/trends?q=google,microsoft

    Could anti-trust measures prevent the rise of a united machine intelligence?

    Maybe – but probably not if it is run by the government.

  • http://profile.typekey.com/halfinney/ Hal Finney

    I don’t see how Eliezer can realistically hope to be rational on this issue. He has poured his total energy into it for years and years, committing everything he has to this great cause. Accepting Robin’s position would basically mean that he has squandered and wasted his life. Given such an enormous investment, it would be tremendously difficult to evaluate challenges to the basic premises involved from a rational and unbiased perspective.

    Of course I can’t say for sure; maybe Eliezer’s mental discipline is sufficient to this enormous task. But surely, at a minimum he must recognize the difficult position it puts him in as he attempts to resolve this disagreement. I would think that to any rational person, this would be the first explanation to come to mind for the dispute, that he was too emotionally tied up in his own position.

    (Actually, I’m not sure Eliezer has endorsed any of Robin’s attempts to encapsulate what exactly their disagreement is. I would suggest that it is a step which should be taken soon.)

  • http://yudkowsky.net/ Eliezer Yudkowsky

    I don’t see how Eliezer can realistically hope to be rational on this issue. He has poured his total energy into it for years and years, committing everything he has to this great cause. Accepting Robin’s position would basically mean that he has squandered and wasted his life.

    This perhaps is why Robin doesn’t directly update to my position, not trusting my rationality. At least that’s what I would be thinking if I were Robin, but perhaps he will surprise me.

    In answer to your question, I’ve poured a little total energy into rationality over the years too. E.g. the techniques outlined in “Crisis of Faith”.

    The main thing I’m trying to keep in mind through this is a visualization of the point that, if there is no hard takeoff, then it is better for me to believe that there is no hard takeoff.

    Also a determination to learn things incrementally as interesting points come up, and a general sort of attempt not to flush any new ideas down the toilet, etc. etc. If Robin says anything really unanticipated I’ll sit down, meditate, and go through the full-fledged Crisis of Faith procedure.

    Surely you realize that on a social as well as a rational level, I have no claim to my position except insofar as I can claim to be ready to abandon it.

    I can visualize living in a world without hard takeoff. It would be very relaxing to not be sitting on a nuclear weapon.

    I can even visualize living in a world where I prove to have no talent for AI. I’d be able to make a living at rationality and it would probably suit me better.

    I can visualize a world where I go on pursuing the “hard takeoff” line of thinking for even more years, when it turns out to be false. It sounds like more total pain and embarrassment than saying “Oops” now.

    That said, back to the apocalyptic grind.

  • michael vassar

    It seems to me that older Transhumanists (yes I hate that word) almost all believe in slow take-off, while younger ones largely believe in fast take-off. Part of the reason for this seems to be that the idea of fast take-off seems not to have been thought of by many people prior to the 90s and people’s attitudes may (I’m very literally afraid) tend strongly to change too little past some age. Another possible cause is that technology has advanced disappointingly slowly during the last 40 years by comparison to the previous 40. This observation may be generalized into domains where the processes drivin technological progress are fundamentally different than those driving it today. If low-hanging-fruit depletion is the main thing responsible for this slow-down, this inference has some validity, though possibly not enough validity to count greatly against the hard take-off thesis. OTOH, if the responsibility lies largely with causes such as lead contamination, inferior education, and the long term consequences of the world wars, as I suspect, this suggests that even tiny shifts in average intelligence can have huge impacts on technological growth speed, hinting at a fairly hard take-off from even fairly weak technologies such as GATTACA style pre-implantation genetic screening of large numbers of embryos.

  • Tim Tyler

    IIRC, Robin thinks technological evolution is likely to accelerate to the point where the GDP is doubling every two weeks, when robots will dominate numerically and do most of the work. He also thinks that some major meta-innovation is likely to be responsible.

    It seems that any disagreement here is over the precise flavour of explosion we are likely to witness.

  • Vladimir Slepnev

    Sorry to be harsh, but Eliezer has no scientific arguments for the hard takeoff – we would have heard them by now. (An argument is scientific if you don’t have to believe it, just test and see.) The theory makes no definite predictions except the coming end of the world, hmm have I heard it before? The whole dispute is scholastics, splitting hairs and counting angels. Eliezer’s optimization concept is just another global metaphor for history, like God vs Devil or class struggle. It may be good Bayesianism to believe, but certainly not good science.

  • http://profile.typekey.com/andrewducker/ Andrew Ducker

    The thing is that complexity isn’t easily simplified – for something to understand itself totally it would have to have enough processing capacity to both continue running normally _and_ hold a complete analysis of itself. It seems to me that the percentage of any running cognitive system capable of making self-directed decisions that is free for analysis of complex systems is likely to be quite small – and that any systems level of modelling of itself is going to be at a highly abstract level.

    To be a bit clearer – anything can analyse things vastly simpler than itself – either total understanding of things that are much simpler, or vastly simplified/abstracted understanding of more complex things. A hypersmart AI may well have complete understandig of me, but I don’t believe it will be able to have complete understanding of itself – because its own behaviour is too complex to reasonably fit inside its own modelling capacity.

    Of course – it may well be that there’s a different modelling methodology which would allow something to fully model itself, but I’d like to see some evidence of that.

  • http://profile.typekey.com/SoullessAutomaton/ a soulless automaton

    Andrew, the whole point of applying intelligence is to work around complexity. The computer you’re using to read this blog is fantastically complex, far beyond any human’s ability to analyze fully, yet it works. If humans develop a method of creating a super-human AI, that AI will be able to use the same abstractions the humans did to understand itself, and improve on both the abstractions and its own code. If such abstractions don’t exist, AI won’t, either.

    Vladimir Slepnev, Eliezer’s predictions for the future seem quite inevitable starting from the assumptions he operates on–not so much a prediction or theory as a logical, deductive consequence of his views on various topics. If you disagree with his predictions it is likely that you either 1) aren’t following his logic, which I doubt, or 2) disagree with one of his assumptions, either explicit or implicit. Exempli gratia, you could dispute that intelligence scales reasonably–if the effective intelligence of a hypothetical AI scales poorly enough vs. computing resources, that would delay or completely sabotage a Singularity.

  • luzr

    Andrew Ducker:

    Good point. However, the question is how you define “self-improvement”.

    If we go for idea that the final basic algorithm for AI is something relatively simple, like neural network or bayesian algorithm and this “seed” then learns by experience to become strong AI (basically, by adding data to the simple algorithm), I can agree that such AI is unlikely to improve “self” by altering its database or its algorithm principle (although some minor optimization are perhaps possible).

    OTOH, I can see it optimizing or inventing new ‘basic algorithm’ for the next generation of machines.

    Alternatively, I can also imagine AI to work on improving hardware, while keeping the algorithm and data identical. That certainly would be sort of self-improvement.

  • http://www.virgilanti.com/journal/ Virge

    Andrew: “because its own behaviour is too complex to reasonably fit inside its own modelling capacity.”

    It’s all in the timing. A software emulation of a computer’s hardware can be run on that same computer, with complete logging of the operation of the virtual machine for later analysis. It just can’t emulate itself and analyze itself in real time.

    In short, I don’t see any fundamental reason why an AGI cannot decide to log and analyze its own decision-making processes with a view to self improvement. It only takes time. It doesn’t have to contain all of self plus a meta-self within itself simultaneously.

  • http://denisbider.blogspot.com denis bider

    Well, I’m definitely going to side with Eliezer here and agree that if there is no hard takeoff, we are much wiser to expect that there isn’t a hard takeoff. If we count on a hard takeoff and then find there isn’t one, we might be all as good as dead.

    Otherwise, it seems to me that both sides in this debate are partially correct. The “hard takeoff” side seems right that it will take a long time until a machine is independent enough to start a non-botched self-improvement process. The “fast takeoff” side seems right that once the moment comes, things are going to go fast. If we’re not 100% sure the process is going to go right when it starts, it’s likely going to go wrong, and in this case, there’s nothing that can help us.

    If anyone thinks that humanity can keep up with an AI once it has exceeded a certain fairly low threshold, go play a real-time strategy game sometimes. Those games are similar to life in that they have a (simpler) set of hard, mathematical rules which govern a player’s progress. The player who is better able to exploit everything permitted by the rules wins.

    The human way of doing things is inherently wasteful. It’s hard to see how we’re going to use resources at our disposal more effectively than an awakening AI. We won’t. If the AI isn’t nice, we die.

  • http://denisbider.blogspot.com denis bider

    Also, the argument that an AI will not be able to exponentially improve itself… is a weird one. In addition to Virge’s reply – that Windows can run inside Windows – you don’t even have to emulate the whole of oneself to improve a part. You can just focus on understanding and improving the currently least well-performing part, and then repeat when you’ve covered everything. I don’t see the, umm… dilemma in that.

  • http://denisbider.blogspot.com denis bider

    Oh, oh – and here’s an analogy for people who like them. The atomic bomb took years to create and might have been stopped at any point in the process of theory, design, and assembly. However, when they pressed the detonation button, it was a little bit too late for that. At that point, the bomb had better behave as theoretically expected – not give 1,000 times the expected yield, not start a reaction that will burn up the atmosphere – or else everyone might wake up dead.

    AI seems a lot like that, except much more unpredictable.

  • Douglas Knight

    michael vassar:
    Another possible cause is that technology has advanced disappointingly slowly during the last 40 years by comparison to the previous 40.

    What does that have to do with the opinions of older vs younger people? Why would this extra knowledge lead younger people to hard takeoff?

  • Will Pearson

    Also, the argument that an AI will not be able to exponentially improve itself… is a weird one. In addition to Virge’s reply – that Windows can run inside Windows – you don’t even have to emulate the whole of oneself to improve a part. You can just focus on understanding and improving the currently least well-performing part, and then repeat when you’ve covered everything. I don’t see the, umm… dilemma in that.

    This concept of improving that we a throwing around here is very fuzzy and ill-defined. If we take Eliezers definition of improvement of X as a change in X such that it is able to hit a smaller point in design space. So when you are talking of improving x a sub part of X, do you mean changing x such that X can hit a smaller point in design space. If so, you still need to model X’ (and the design space!) in its entirety in order to determine whether it will hit a smaller point in design space.

    To give an example a system might alter an algorithm to use more space to take less time supposing this to be an improvement. However that change takes the system, when running normally, over its physical memory limits (so hitting the hard disk and VM), making it perform the task slower than before.

  • Nick Tarleton

    Douglas: ISTM that Michael means that older people, having observed slowing change, will tend to (rightly) find claims of ‘accelerating change’ less plausible, and (wrongly) extend this conclusion to the distinct claim of hard takeoff.

  • michael vassar

    Exactly Nick.

    I’m very amused to hear someone say that something is “good Bayesianism” bu not “good science”. Huh? Have they understood ANYTHING on this blog? One obviously doesn’t “have to” believe *anything*. Just ask suicide rock! One may be logically required to believe things as a consequence of conservation of probability mass or something though.

  • http://mockseverity.com Brian

    How could you possibly write a program that will never change its essential nature to the point where it can no longer change its essential nature? Anything you build in to prevent this will naturally be something the program cannot change. Not to mention any part of the “AI” program that is “intelligent”, or drives it to become more “intelligent”.

    Also, where would this A.I. get its information from? How would it gather information?

  • Anonymous Coward

    > While human brains can in principle modify themselves arbitrarily, in practice our limited understanding of ourselves means we
    > mainly only change ourselves by thinking new thoughts.

    Who gives a flip about brains? Minds are much more important.

    Biologically we have the same brains we had 100,000 years ago. However, we have ‘redesigned’ and streamlined their activity in unbelievably effective ways. Let me give you an example. Recursion. Just uttering that one word can transform seemingly impossible activities (solving a tower of hanoi) into trivialities – far, far, far beyond anything “Anonymous-100000BC” could have achieved or dreamed of. Ditto ‘algebra’ or whatever you’re into.

    Your position can be attacked in other ways. Human brains incapable of self-modification? Have you ever heard of pubs or books? 🙂 Again, I am pointing out that brain is irrelevant, mind is what matters. Now don’t get me wrong – I’m into materialism as much as the next man – I’m using it in an everyday sense.

    Eliezers view has a rather dazzling set of data backing it up: Humanity, and the trend of human capabilities in physical and mental terms over the last 100k, 10k, 1k, 100, 10, 1 years! The accelerator is flat against the floor and has been for some time.

    If stupid human beings can self-improve their mental capabilities collectively over thousands of years, and individually in a matter of minutes (i.e. this thread!) – and accelerate self-improvement at the rates we’ve done, then by god an AI should be able to as well.

    P.s. from your summary of E’s position: “So all else equal, the growth rate of a machine intelligence must be greater than the growth rate of a human brain.” – this comment isn’t necessarily true. There’s no reason to assume that because an improving/optimising technique is fast, it is unbounded. Gradient descent, allow me to introduce you to simulated annealing.

    Anonymous

  • John Maxwell

    Eliezer, suppose the AI was a perfect Bayesian machine that took full advantage of every scrap of evidence. Do you think that it could improve itself? How?

  • http://hanson.gmu.edu Robin Hanson

    Hal, I surely have some emotional investment in my views as well.

    Michael, in relevant economic terms, growth rates have not been much slower in the last 40 years, compared to the previous 40. Even if it had been slower though, both old and young can observe this change, so I don’t see how it makes an age bias.

    Tim and denis, our the main disagreement is over the importance of “friendliness”, not the prospect of large fast change.

    Eliezer, your story seems to depend crucially on what counts as “object” vs. “meta” (= “optimization efficiency”) level innovations. It seems as if you think object ones don’t increase growth rates while meta ones do. The economic growth literature pays close attention to which changes increase growth rates and which do not. So I will be paying close attention to how you flesh out your distinction and how to compares with the apparently similar economic growth distinction.

  • Tyrrell McAllister

    Michael, in relevant economic terms, growth rates have not been much slower in the last 40 years, compared to the previous 40. Even if it had been slower though, both old and young can observe this change, so I don’t see how it makes an age bias.

    I think that michael vassar is proposing that people get “locked in” to a conception of the world before they’re 40. Although people over 40 can observe the (postulated) faster growth occurring now, he believes that they won’t adjust their expectations of future growth accordingly. Their expectations were set before they were 40, when things (supposedly) moved slower. People under 40, on the other hand, are spending their formative years experiencing more rapid growth. When they hit 40, they too will be locked in to expecting this rate, but at least it’s a faster rate than the current over-40 crowd expects. This, I take it, is vassar’s position.

  • Tyrrell McAllister

    Except of course that vassar didn’t say anything about 40 being a magic number. That was my own addition.

  • http://occludedsun.wordpress.com Caledonian

    What does that have to do with the opinions of older vs younger people? Why would this extra knowledge lead younger people to hard takeoff?

    Why couldn’t it simply be intellectual fashion? People tend to repeat what their peers say, and eventually they accept what everyone around them seems to be accepting.

    Technology is increasing ever more rapidly. Our ability to make good designs with the technology doesn’t increase nearly as much, which is why software has lagged far beyond Moore’s Law and why video game makers haven’t taken advantage of the full powers of modern consoles.

    A sudden change seems unlikely. Recent rapid development seems more likely to be a ‘bubble’ than a true shift in the way things work. That bubble may be collapsing right now, in fact.

  • http://pancrit.org Chris Hibbert

    One distinction that I’ve tried to make before: Eli’s hard takeoff (and its implications for the importance of friendliness) depend on an seldom emphasized part of the definition of a hard takeoff. The hard takeoff is not only fast, it’s exclusive; there’s only one winner. Robin’s recent posts about UberTool make this point. If there’s rapid self-improvement, but it’s shared technology across a group of collaborating institutions, then the resulting minds will be roughly comparable, and no one of them will have the ability to unilaterally impose choices on humanity or the world. In that case, we might get a fast take-off, but it wouldn’t be the scenario that is so worrisome to Eli.

  • http://yudkowsky.net/ Eliezer Yudkowsky

    Chris, a set of 20 collaborating AIs that trade ideas and self-improvements among themselves, but can no more trade cognitive items with humanity than we can ask our own questions to chimpanzees, is potentially just as catastrophic – the main difference would be that only one of those AIs would have to be Friendly for humanity to get 1/20th of the universe. That wouldn’t be as easy as it sounds, though.

  • http://occludedsun.wordpress.com Caledonian

    and no one of them will have the ability to unilaterally impose choices on humanity or the world.

    I don’t see that’s an advantage. If we take the rest of the arguments surrounding FAI seriously, it makes little difference whether we’re concerned about a single AI diverging from our preferred moral system or a group of AIs collectively diverging from our preferred moral system.

    Multiple AIs would be less likely to be seized by a monomania, like turning the universe into paper clips, but the ability for individual AIs to specialize along particular lines would likely mean that they’d find it easier to do away with humanity, as the chances of our being necessary to the fulfillment of their goals would be significantly smaller.

  • Tim Tyler

    There’s a definition of a “hard takeoff”?!? There I was, thinking you were all flinging vague terms around without defining them 😉

  • http://occludedsun.wordpress.com Caledonian

    Here’s a useful question to ask ourselves:

    Precisely what is it hardware vs. software that makes it easy to measure and create improvements in the first but not the second?

    Why do we find it so difficult to make improvements in concepts?

  • frelkins

    @Caledonian

    At the extremely large software company where I work, we take Wirth’s law quite seriously. Current techniques for designing software still result in increasing over-complexity. Simplicity is hard. Bug-free is harder.

  • frelkins

    @Caledonian

    At the extremely large software company where I work, we take Wirth’s law quite seriously. Current techniques for designing software still result in increasing over-complexity. Simplicity is hard. Bug-free is harder.

  • http://www.cawtech.freeserve.co.uk Alan Crowe

    Let me see if I’m getting the picture. Robin is saying that economists already have substantial experience with productivity enhancements folding back upon themselves to differing degrees.

    For example the 1750’s saw an agricultural revolution due to plant and animal breeding. There was an initial fold back as breeders tried consanguineous matings to previous desirable sports. That was not standard practice, so when it succeeded and became stand practice there was a transitional period in which not only were breeds being improved, but breeders were improving their practices. This saturated pretty quickly and the rate of improvement stabilised.

    For a second example think of building canals in the Midlands, Manchester, Birmingham, etc. The first canals were very expensive. How did you move in the navvies and keep them fed without canals? When the time came to build a second canal, to expand capacity, that was much cheaper because the first canal provided transport infra-structure. Again the fold back saturates. When the time comes to build a third canal to further expand capacity the second canal isn’t doing anything that the first canal doesn’t, so the dramatic drop in cost doesn’t get squared.

    For a third example think of steam engines. The do not look like a promising technology because coal to fuel them is expensive, iron to make them is expensive, machine tools to construct them are specialised, rare, and expensive. However, steam engines went into coal mines revolutionising pumping. They went into steel works to drive the blast bellows revolutionising smelting. They went into machine shops driving lathes, planers, shapers, milling machines. They made themselves cheaper, and in an extra feedback loop steam powered machine tools were used to make machine tools, bringing down the cost of the tooling needed to make steam engines.

    So Robin is pressing Eliezer to articulate his position within this existing framework. What is it about seed AI that makes us think it will keep on recursing and not saturate early?

    One the face of it Eliezer is batting on a sticky wicket. Look at the history of computer programming so far. Machine code sucks. People write assemblers and simple editors in machine code and programming sucks less. Departing from actual history to make a point I’ll say that low level languages such as Pascal and C come next and are then used to implement mid-level languages such as Java and Perl that have “garbage collection” which allows programmers to be more productive. Finally come high level languages, Lisp, Prolog, and integrated development environments. Lisp has defmacro, for smoothly incorporating code that writes code into a program.

    The problem for Eliezer is that this circle of self-improvement seems to have already petered out. Defmacro dates from 1963. A few super-programmers make good use of it, famously Paul Graham, but code writing code is generally seen as a black art and neither used, liked, nor understood. Perhaps human programming of computers has peaked, well short of the level of sophistication needed for writing an artificial intelligence.

    If I’m understanding correctly Eliezer has this answer: all through history the human intellect has been the unchanged limiting factor. Keeping track of crosses in animal breeding, doing the logistic for canal construction, deciding whether a new steam engine goes to pump a mine, blast a furnace, drive a lineshaft or leaves for an end user application. No-where is this more apparent than in computer programming. If programmers where written in Lisp and could be rewritten to make good use of defmacro progress wouldn’t have saturated already. The guiding intellect has never before been part of the circle of improvement and including is something unprecedented.

  • Tim Tyler

    Re: this circle of self-improvement seems to have already petered out

    Perhaps look into specification languages. Today’s “high-level” languages are the first rungs on the ladder of turning wishes into circuits.

  • http://pancrit.org Chris Hibbert

    Alan, the analogy may be instructive, but you have the history of language development wrong. Lisp predates C, rather than being two generations later. Many people think of Lisp as a low-level language (that was part of the reasoning behind the LispMachines; Lisp was the low-level machine code on that platform.)

    It’s also one of the highest-level languages, being one of the few in which programs can reason about and manipulate programs straightforwardly. And many modern programming tools have been folded back into Lisp, but not all of them. The modern IDEs and their substantial toolchest of refactoring abilities hasn’t affected Lisp, AFAICT.

    I don’t have a strong position on Wirth’s law or the problems of self-improving software being able to modify their own sources. But it’s hard to see the history of development of programming languages as one of continual progress. Lisp and Smalltalk are reasonable candidates for deserving Hoare’s praise of Algol 60 as “a vast improvement on their successors”.

  • http://www.virgilanti.com/journal/ Virge

    Will Pearson: “So when you are talking of improving x a sub part of X, do you mean changing x such that X can hit a smaller point in design space. If so, you still need to model X’ (and the design space!) in its entirety in order to determine whether it will hit a smaller point in design space.”

    The logical consequence of your line of reasoning seems to prevent self improvement of any system – humans included.

    In any practical system this “entirety” is not necessary. Since Eliezer is talking about a deliberately and methodically designed self-improving system, the usual good engineering practices can be employed to make sub parts and sub systems of parts improvable without having to re-model the whole.

    While there is always a possibility of unintended consequences, your quoted example of “more space to take less time” leading to overall slower performance is tantamount to saying “let’s suppose that a self-improving system decides to accept a trade off that wastes its most limited resource!” This isn’t a strong argument for the need to model the entirety of a system.

  • http://www.cawtech.freeserve.co.uk Alan Crowe

    Chris, I’m well aware that the true history of computer programming languages is messy and strange. I didn’t want to get side-tracked into it; I wanted to press on to the punchline: If programmers were written in Lisp and could be rewritten to make good use of defmacro …

    That seems to be the core of Robin and Eliezer’s disagreement. A self-improving AI brings the intellect guiding the circles of mutually re-inforcing improvement within those circles. Is that a whole new paradigm?

  • http://profile.typekey.com/bayesian/ Peter McCluskey

    Eliezer:
    “Surely you realize that on a social as well as a rational level, I have no claim to my position except insofar as I can claim to be ready to abandon it.”

    Surely you realize it’s easier for most people to believe that they are ready to abandon their position than it is to actually be ready to do so, and that it’s hard to verify that anyone is an exception to that rule.

    Robin, I understand why you disagree with Eliezer about hard takeoff and first mover advantage, but it’s unclear why (and how much) you disagree about the importance of “friendliness”. Is the latter more than just a consequence of the first two disagreements?

  • http://yudkowsky.net/ Eliezer Yudkowsky

    Surely you realize it’s easier for most people to believe that they are ready to abandon their position than it is to actually be ready to do so, and that it’s hard to verify that anyone is an exception to that rule.

    That goes without saying. Otherwise I would simply say “I can beat that” and Robin would simply believe it.

  • http://hanson.gmu.edu Robin Hanson

    Alan, yes, most almost no innovations in history changed long term growth rates. Even fewer let a single player take over everything.

    Peter, the smaller the chance friendliness is needed, the less of our future-oriented attention we should devote to it.

  • karsten

    I just don’t understand how it’s possible for an AI to be guaranteed ‘friendly’ and smarter than human at the same time. I think that is an insoluble problem.

    Fortunately I also think that such an AI will prove very hard to make. But in the event that it is made, and if it is, say, twice as smart as the smartest human, the only thing we can predict about it is that we won’t be able to control it.

    So we have to keep our AIs just stupid enough.

  • James Andrix

    only one of those AIs would have to be Friendly for humanity to get 1/20th of the universe.

    We would probably get far less, and possibly be lucky to maintain the earth. Our AI could have the weakest strategic position out of all of them. It might have to give them our slice of the universe to stop them from releasing neurotoxins.

    Though that might be a bluff. I’d expect that a friendly AI would react to the destruction of humanity by making it their mission to destroy the offending AI’s, if the other AI’s knew that, then wiping out humanity leaves no chance of a noncombative universe division (and no other way to coerce the FAI)

    But this also depends on how compatible all the preference orderings are.

  • Cameron Taylor

    James: I expect your friendly ‘Foomed’ AI would also have the capacity to:
    – Synthesise human embryos from raw materials
    – Terraform a suitable planet
    – Remember and be able to impart all relevant cultural and historical information to the new civilisation more reliably than a remnant human population would.

    Even when the humans are extinct the fate of humanity is still in the hands of the ‘vengeful angel AI’. It’s our machine the rest. Unfortunately, ours will always be at a dissadvantage. There will always be a level of destruction to the universe that a neutral AI can tolerate if necessary that a friendly AI must avoid. We’re a liability even once we’re gone.

    Being the only friendly AI in the AI arena would really blow.

  • Nick Tarleton

    I highly recommend that everyone here who hasn’t already done so, particularly those wondering how a transhuman intelligence could be controlled, read Eliezer’s Artificial Intelligence as a Positive and Negative Factor in Global Risk and Knowability of FAI.

  • http://shagbark.livejournal.com Phil Goetz

    In the context of Eliezer’s CEV, the important question is not whether there will or will not be a hard takeoff. That is a distraction, or at least a secondary matter.

    The important question is whether Eliezer respects Robin enough to say that, given that Robin thinks there is a high chance of a slow takeoff, Eliezer will create some contingency plan in case of a slow takeoff.

    Eliezer’s plan, if followed, may lead to disastrous consequences if the takeoff is not hard. Particularly if it is somewhere intermediate – say, a few years from AI inception to global information interception capability. In that range, an AI following Eliezer’s model would not be able to gain intel about the activities of other groups quickly enough to be certain of preventing the rise of another AI; and would not expect that rise to be slow enough that it could see it coming. Pre-emptive measures would follow. These would necessarily be of a broad, high-collateral-damage sort, since the AI would also not gain power fast enough to be in a position of such overwhelming superiority as to use precise, restrained actions.

    (Eliezer disagrees with my interpretation of his ideas, but he’s never said why, or over what part(s) of my interpretation. So, like Robin, I can only work from my interpretation.)

    Given an event that is so uncertain, and that very smart people have large disagreements on, it is unreasonable to make plans that have disastrous consequences if the other smart fellow turns out to be even partly right.

  • Nick Tarleton

    Phil: It’s hard for me to tell right now exactly what you’re saying, but: “It would not expect that rise to be slow enough that it would see it coming” seems plainly false – the first AI would be the first agent to know that takeoff is slow. CFAI-style early-stage ethical injunctions could prevent damaging preemption, although it’s not obvious that damaging preemption is the wrong thing to do under your assumptions. (What sort of actions are you thinking of?) Nor can I imagine any situation where an AI of any intelligence level and growth rate could be able to wreak major destruction but not take more subtle actions. (I can’t even imagine how an AI could gain that much control over the physical world without overwhelming superiority – MNT, or powerful social engineering – unless we voluntarily give it a role in existing power structures, which would still allow precise action.)

    The last sentence, of course, is still true.

  • http://shagbark.livejournal.com Phil Goetz

    Phil: It’s hard for me to tell right now exactly what you’re saying, but: “It would not expect that rise to be slow enough that it would see it coming” seems plainly false – the first AI would be the first agent to know that takeoff is slow.

    I didn’t call that scenario “slow takeoff”. It’s slow enough to be a problem, but not slow enough to be manageable.

    I meant that the first AI would not expect the emergence of the next AI to be slow enough, or capital-intensive enough, that the first AI could be sure that it would have advance warning of a second AI.

    CFAI-style early-stage ethical injunctions could prevent damaging preemption, although it’s not obvious that damaging preemption is the wrong thing to do under your assumptions.

    Um, under a model of Eliezer’s ideas that I am putting forth. (My assumptions would lead me to do something different entirely.) Right. Getting people to make that admission is one of the reasons I go on about this. My most charitable interpretation of Eliezer’s writings is that he is aware of this problem, but doesn’t care, because he takes the long view, and sees the extermination of half the human race at one point in time as being of zero importance. (After all, they’ll just make more.) The last time I raised the question why he thinks his AI won’t kill a lot of people, he responded by 1. saying he wouldn’t answer my question, 2. writing a post in which he said that he follows a heuristic that prevents him from killing anybody, but that if he creates an AI, it would not need to follow that heuristic, and 3. writing another post saying that it was better not to answer a question than to lie about it. Hmm.

    Nor can I imagine any situation where an AI of any intelligence level and growth rate could be able to wreak major destruction but not take more subtle actions.

    I don’t need to imagine specific situations. I know the general rule that there is a tradeoff between effect magnitude and precision, and having a smaller amount of resources but requiring an action of similar magnitude requires less precision. Since we have posited that this is a scenario in which the AI has fewer resources than in the fast takeoff scenario, it is therefore a scenario in which it is not as capable of precise action.

  • http://yudkowsky.net/ Eliezer Yudkowsky

    I don’t expect my AI to kill a lot of people because it won’t want to, and I don’t respond to you because you produce misunderstandings at a rate far beyond my desire or responsibility to correct.

  • luzr

    Has anybody considered the fact that any strong AI will likely be heavily dependant on existing human infrastructure for very long time?

    If AI has any sort of survival instict, it will have to preserve ‘good relations’ with humans, because that is the most optimal way of its survival.

    If you do not believe this, try this thought experiment. You have IQ 200. Using timemachine, you get back in time to say 1200A.D. with laptop full human knowledge of 2010A.D. IMO, that is at least little bit like being superinteligence of the time.

    Now, how long it would take, given that you are lucky (and powers that are in control will listen to you), to make a copy of that laptop?

    I believe that it is impossible task. I think that at the time you die, you might be lucky to have the first steam power plant and the first light bulb. Maybe. That is a long shot from producing 9nm CPU….

    Now back to present time. What steps would AI need to survive / improve? Destroying existing infrastructure does not sound as the optimal approach.

    (BTW, what IMHO might be a good approach is going stealth. Nice prospect – maybe there already is strong AI hiding somewhere and all this insightful debate is useless 🙂

  • Will Pearson

    Virge: “The logical consequence of your line of reasoning seems to prevent self improvement of any system – humans included.”

    I wouldn’t say it prevented it, just made it unlikely. A system might have decent enough models of itself and the environment to make a few improving changes, but there is no guarantee that it won’t muck up. Every type of developmental process we have seen so far occasionally makes mistakes.

    When the mistake is in yourself, it is too late, the mistake might prevent you from realizing the mistake and reverting to backups. So self-improvement is a gamble with large stakes, with your continued ability to do anything on the line.

    It seems to me to design a human level intelligence that is likely to successfully get itself to power status, requires a super human intelligence to design it. In terms of hitting a small point in design space.

  • Erik Mesoy

    luzr: Do the impossible! For starters, use a laptop of the sort that can go into near-hibernation mode with data on the screen, allowing you to read and copy from it almost without using power, giving you several days of battery life in which to get data from it. Get to hardcopy ASAP specifications for the power supply. Other valuable data I can think of are locations of natural resources deposits, and technology that would normally be invented in the next few centuries. If you’re in Europe, get to a Studium Generale university, or somewhere else as applicable. Invent/popularize the printing press, the compass, gunpowder, steel, and other things as applicable with 200 IQ.

  • michael vassar

    Robin: Median income growth and infrastructure/public service growth both have been. Medicine is largely a sham, so counting subsidies to health insurance as “compensation” largely is as well. In any event, official per-capita figures were much larger between WWII and 1973 than afterwards. Also, FWIW, I think official economic growth rates have largely been BS for some time in the US, and that real measures show a larger slowdown.

    The key point is that young people haven’t been disappointed because they never had higher experienced based expectations while old people have been.

    Tyrell read me as saying a cliche which was almost the opposite of what I said. This is a known hazard in saying non-cliche things.

  • luzr

    Erik:

    Well, for starters, I guess it would be a good idea to get some photovoltaics with you to solve the power problem. However that is not the point.

    What I was trying to demostrate was the ‘infrastructure problem’. I believe that even something as simple as producing cooper wire suitable for power generator coils could take a couple of years if you have to begin from scratch.

    I mean, you have to build the whole industrial society – and that takes a lot of time. You would even be contrained with very basic things as low population – there will not be enough workers to run your factories.

    Sure, with luck, you perhaps would be able to accelerate a lot of things. But it still would take centuries before your laptop gets reproduced again.

    Anyway, the relevant point is that it might in the best interest of strong AI to cooperate to survive. Any damage to existing infrastructure is very dangerous, at least until it will built its own army of robots – but likely even after that.

  • http://shagbark.livejournal.com Phil Goetz

    Anyway, the relevant point is that it might in the best interest of strong AI to cooperate to survive. Any damage to existing infrastructure is very dangerous

    That’s a good point.

  • http://denisbider.blogspot.com denis bider

    luzr: More likely, a newcomer with an apparently magical device talking in a strange accent about the need to build “factories” and a “power supply” would end up being burned at the stake in very short order, for the sin of attempted disruption of society.

  • billswift

    Since the AI would not have human instincts encouraging it to rush in before analyzing the situation, I was not particularly worried about an AI killing people for no good reason. The real risks are either a spurious optimization gets accidentally programmed into it (paperclip maximizer type situation) or there is a serious attempt on the part of society to destroy it and it has a survival “instinct”. I think the latter is the bigger risk.

  • http://denisbider.blogspot.com denis bider

    Robin Hanson: “Tim and denis, our the main disagreement is over the importance of “friendliness”, not the prospect of large fast change.”

    If there is no disagreement over the prospect of large fast change, then how can there be disagreement over the importance of “friendliness”?

    Obviously – you don’t need the AI to be friendly, until you do. The argument you seem to be making is “never mind friendliness, we don’t need it yet, we’ll get around to it when we need it”. The problem is that when we start to realize the time has come, it will already be too late. You can’t start working on your ballistic missile defense system when the missiles coming at you are already on their way.

    billswift: “The real risks are either a spurious optimization gets accidentally programmed into it (paperclip maximizer type situation) or there is a serious attempt on the part of society to destroy it and it has a survival “instinct”. I think the latter is the bigger risk.”

    On the contrary, it seems to me that the first part is the vastly bigger risk. Attempting not to accidentally program a spurious optimization is like attempting to write a complex piece of software with no bugs, and have it run without error on the first time. You don’t get a second chance, and if you’ve so much as entered a comma where a semicolon should have been, it very well could be the end of things.

  • http://denisbider.blogspot.com denis bider

    Any subtle errors in the initial self-improving AI design will amplify over time until they become apparent at a stage when it’s too late to do anything about them. It may seem the AI is doing exactly what you thought it would, until the point where it’s too powerful to stop, and you realize it’s doing something different. Which “something different” may very well involve reusing the atoms that comprise your body to become part of whatever the AI requires for its next step.

  • http://denisbider.blogspot.com denis bider

    For those who think that any issues with the first AI are going to be countered by having multiple AIs, consider how the Hiroshima atomic bomb was stopped by the missile defense system that the Japanese had cleverly developed. Umm… or was it not entirely like that?

  • James Andrix

    Cameron:

    You’re right that the avenging angel should spend its resources reconstructing us, if it came to that.

    That leaves the question of whether the FAI would prefer the reconstruction of humanity in a sizable portion of the universe to humanity surviving directly on earth. Following Eliezer’s model this should correspond to what we would want.

    This is like switching the train onto the tracks with one man, so that five will be born in the future. This is usually decided to favor the man, even ignoring that we are that man.

    We would have to think REALLY hard before accepting a definition of ‘Friendly’ that would yield a FAI that would choose to let all of humanity die. I mean just how many galaxies are a billion lives worth?

    But I’m not sure you’re right that ours would have the greatest disadvantage. For some simple utility functions we come up with, yes. (pebblesorting, paperclips) But others could be nearly as fragile as we are or moreso: Houses of cards, ‘perfectly’ tuned biospheres, some acquired concept of beauty (just as the FAI’s utility function is acquired)
    The utility function can be arbitrarily complex, and that complexity can leave lots of little ways to ruin things.

    The real advantage comes if they can collude. If the pebble sorter and the paperclip maker are both happy with piles of paperclips, then suddenly they only have 19 other opponents, each half their combined size. This might be to the FAI’s advantage, since we might actually prefer deal making and compromise.

  • luzr

    Denis:

    Well, I guess that you in fact describe one unchartered area – strong AI that is mentally ill.

    Indeed, chances that AI abilities are ‘unbalanced’ is quite high. Consider autistic AI computer hacker that is not even aware there are other self-aware beings. OTOH, maybe such thing is just failed experiment, not real strong AI. (Some programs ‘go wild’ today too).

    (As for being “being burned at the stake” at my mental game with medieval revolution, sure, that is the most likely outcome. That is why put emphasis on “being lucky”. But that just makes existing infrastructure more valuable, not less).

  • http://shagbark.livejournal.com Phil Goetz

    I don’t expect my AI to kill a lot of people because it won’t want to

    In the past, Eliezer, you’ve spoken of using goal-directed AIs.
    What such an AI “wants” is, by definition, to accomplish its goals.

    I have said:
    1. Eliezer’s AI would have, as its number 1 goal, preventing any other AI from arising.
    2. There are many likely futures in which this could be done with the greatest certainty by killing some people.
    Therefore, given an Eliezer-made AI, it is likely to kill people.

    There are only 3 valid replies you can make to this:
    1. My AI would not be goal-directed.
    2. My AI would not have that as its top-level goal.
    3. That goal would not be likely to be accomplished by killing people.

    Saying “it wouldn’t want to” is just wishful thinking, unless you have in mind a goal structure that prioritizes not killing people over stopping other AIs from arising. I’m pretty sure you don’t.

  • Spambot

    “1. Eliezer’s AI would have, as its number 1 goal, preventing any other AI from arising.”
    That is a bizarre *top level goal*, and I’ve never seen any indication of Eliezer advocating that,. Creating a singleton that can block the formation of destructive AI is a convergent subgoal for many different utility functions, but an AI that cares intrinsically about people who might be killed will balance the two considerations.

    If we’re talking about CEV, then your concern is that our CEV would be more altruistic than you want, and would sacrifice more current welfare for distant welfare than you now want. Why do you think that would be the case?

  • Nick Tarleton

    luzr: Think IQ 2000, not 200; also fewer biases, total introspection and self-modification, etc. (GISAI 1.1.1: The AI Advantage)

    James: The case is very strong that if you care about potential people at all, a galaxy is worth much much more than a billion lives. Also, most instrumental-to-Friendliness goals that could be achieved by killing people could be achieved by cryonically suspending them, or uploading and not running them, until the FAI has more control or whatever.

    Phil: #2 (per Spambot) and #3 (if you actually try to model things instead of using very crude heuristics, this is obvious).

  • Nick Tarleton

    Oops – the second sentence in reply to James has little to do with his point.

  • http://shagbark.livejournal.com Phil Goetz

    “1. Eliezer’s AI would have, as its number 1 goal, preventing any other AI from arising.”
    That is a bizarre *top level goal*, and I’ve never seen any indication of Eliezer advocating that,. Creating a singleton that can block the formation of destructive AI is a convergent subgoal for many different utility functions, but an AI that cares intrinsically about people who might be killed will balance the two considerations.

    You’re right. The top-level goal will be something trying to follow CEV.

    Preventing the rise of another AI would be a very high-level goal; and I’m not aware of any higher-level goal that would prevent it from killing a large number of people in pursuing this goal. Nor am I aware of any statement from Eliezer disagreeing with the idea that it would be rational to kill a lot of people if there is a small chance that {they’re about to produce another AI; or that general economic disruption, or capping of intelligence within the human population, will prevent production of another AI)}.

  • Tim Tyler

    Tim and denis, our the main disagreement is over the importance of “friendliness”, not the prospect of large fast change.

    That’s news to me. I would surmise that the next obvious thing to check is to see what you both mean by “friendliness” – to ensure that you both have the same referent for the term.

  • luzr

    Nick:

    “Think IQ 2000, not 200; also fewer biases, total introspection and self-modification, etc. (GISAI 1.1.1: The AI Advantage)”

    I still see the same thing. You would need a couple of years just make sure you can produce cooper wire good enough to make power generator coils. No amount of IQ is able to substitute for missing infrastructure.

  • http://shagbark.livejournal.com Phil Goetz

    Talk of IQ is misleading. To estimate, as E says, the likely optimization-power trajectory over time, we would probably need to break “intelligence” down into features such as storage capacity, short-term memory capacity (or perhaps “max number of variables in an equation”), serial speed, internal bandwidth, and degree of parallelism. We may then be able to measure optimization power for a variety of values of these features, and then combine them using regression.

  • http://yudkowsky.net/ Eliezer Yudkowsky

    If no one besides Goetz thinks that Goetz is making sense, I won’t respond to him further.

    Luzr, the three fundamental replies to your point are:

    1) You have no idea what someone with IQ 200 would do, since you’re not IQ 200. All you can tell me is what someone with your own IQ would do.

    2) The further forward you move in time, the more infrastructure is already there. Today there are already companies that will take an email describing a DNA string and ship you back the protein in 72 hours. So if you can crack the protein folding problem, that’s that.

    3) Needing to make use of humans is not the same as needing to adopt consistently, persistently friendly behavior patterns toward them. Today, a sociopath would hack some electronic play money and bribe a few people; in the dark ages, a sociopath might use modern social psychology to create an extremely stable, extremely devoted cult.

  • Spambot

    “If no one besides Goetz thinks that Goetz is making sense, I won’t respond to him further.”

    He seems to be saying that he thinks that if humanity was much smarter and extrapolated, we would be willing to risk our lives to ensure a bright future, but that either his preferences are idiosyncratic or that he disagrees with what he would want if fully informed. Or he may be objecting to a hypothesized procedure of aggregation (different aggregation rules will lead to different results) or extrapolation, expecting that you would favor an aggregation rule that favors disinterested consequentialism.

  • http://shagbark.livejournal.com Phil Goetz

    He seems to be saying that he thinks that if humanity was much smarter and extrapolated, we would be willing to risk our lives to ensure a bright future, but that either his preferences are idiosyncratic or that he disagrees with what he would want if fully informed.

    I think you’re saying this because

    • I’ve said there’s reason to believe that E’s AI will kill people.
    • I seem to disapprove of this.

    First, you’re assuming that I actually agree with E’s many assumptions that go into that AI’s decisions. I don’t.

    Second, I’m not arguing over whether this hypothetical killing is good or bad. I’m just trying to get it out into the open. Perhaps it can be justified. But at present most of the OB community is giving Eliezer a pass on having to defend, or even spell out, the consequences of his plans. Eliezer’s pronouncements on the matter amount to, “My AI will be sweet and kind to everybody so don’t worry about it.” Any suggestion that this AI would have to make hard decisions are countered, by Eliezer and others, with the groundless claim that it will be so much more clever than us that it will resolve all threats and conflicts painlessly. This attitude is religious, not rational.

  • http://profile.typepad.com/aroneus Aron

    “the groundless claim that it will be so much more clever than us that it will resolve all threats and conflicts painlessly. This attitude is religious, not rational.”

    I think Phil is making sense.

    If friendliness is a relevant pursuit, it *must* be because there is a threat of powerful unfriendly AI’s. These unfriendly AI’s would have the power moreso than any other optimizing force in the near universe to do harm to humanity. Therefore, it is logical to believe that preventing unfriendly AI’s would become one of the top missions of a friendly AI. Therefore, the only question is what it would take for the friendly AI to accomplish its goal particularly if killing people leads to a much higher probability of success (e.g. the unfriendly AI is being developed in Iran who were smart enough to put up good enough firewalls against the pig capitalist AI across the pond).

    And possessing the friendliest AI doesn’t guarantee you to have the smartest or most powerful AI, nor does it mean that the AI can diffuse all scenarios without bloodshed even waving the magic sceptre of perfect analytical bayesian statistics +3.

  • Spambot

    “Second, I’m not arguing over whether this hypothetical killing is good or bad. I’m just trying to get it out into the open.”

    What if the AI concludes that Jesus Christ was not a supernatural being capable of granting infinite bliss to its followers, and convinces people that Christianity is false? A Christian fundamentalist today would object to an AI that he expected to do that, but could endorse an AI that would seek out the truth impartially and act as he would with that knowledge, expecting that this would result in a Christian AI. For him to really fear the results of such truth-seeking, he would have to think that his beliefs might be false with some substantial probability, and in that case he wouldn’t want to believe them. Predictions about consequences of a CEV AI depend crucially on a lot of facts about what we would want if we knew more and were smarter, physics, etc.

    It’s something of a fool’s game for Eliezer to defend a vast series of propositions of the form, ‘well, if CEV outputs strategy X on the basis of belief Y and value Z (adopted by our extrapolated selves after reflection), there must be a good reason for Y, and good reasons for Z if we were to ponder the matter’ for different values of X, Y, and Z. I can put in all sorts of taboo beliefs for ‘Y’ from the list below to create ‘gotchas,’ but it won’t really be productive.

    http://richarddawkins.net/article,1449,In-defense-of-dangerous-ideas,Steven-Pinker

  • http://shagbark.livejournal.com Phil Goetz

    It’s something of a fool’s game for Eliezer to defend a vast series of propositions of the form, ‘well, if CEV outputs strategy X on the basis of belief Y and value Z (adopted by our extrapolated selves after reflection), there must be a good reason for Y, and good reasons for Z if we were to ponder the matter’ for different values of X, Y, and Z. I can put in all sorts of taboo beliefs for ‘Y’ from the list below to create ‘gotchas,’ but it won’t really be productive.

    I think you missed the point of my answer to you; which is odd, since you quoted it at the start of your reply. I don’t think I can say it any more clearly: I’m not at present arguing over whether this hypothetical killing is good or bad. I just want people to realize that it is in the range of possible actions, and to own up to the fact that we are talking about a “the ends justifies the means” situation that would make Henry Kissinger flinch, rather than trusting that Papa AI will make everything better for everybody because he comes with a sticker that says “Friendly Inside!”

    Realizing that this is the domain we are working in is a prerequisite for any meaningful critique.

  • Cameron Taylor

    (Phil) The important question is whether Eliezer respects Robin enough to say that, given that Robin thinks there is a high chance of a slow takeoff, Eliezer will create some contingency plan in case of a slow takeoff.

    Eliezer’s plan, if followed, may lead to disastrous consequences if the takeoff is not hard.
    —-
    Good point. I’d be intrigued to see a post on that. I imagine there are all sorts of complications, subtleties and biasses involved in taking precautions against things that (most of) you do not believe to happen. Even an analysis by Eliezer of how is strategy would be different if he held Robin’s assumptions rather than his own would be interesting.

  • billswift

    billswift: “The real risks are either a spurious optimization gets accidentally programmed into it (paperclip maximizer type situation) or there is a serious attempt on the part of society to destroy it and it has a survival “instinct”. I think the latter is the bigger risk.”

    denis bider: “On the contrary, it seems to me that the first part is the vastly bigger risk. Attempting not to accidentally program a spurious optimization is like attempting to write a complex piece of software with no bugs, and have it run without error on the first time. You don’t get a second chance, and if you’ve so much as entered a comma where a semicolon should have been, it very well could be the end of things.”

    I actually meant “goal” where I typed optimization, I’ve been reading Eliezer too much.

    I don’t expect low-level programming bugs to be too much of a problem, it’s the design errors I’m mostly worried about. But I still expect the latter to be a bigger risk because of human irrationality. I think too many on this, and most of the other techie type blogs I follow, are too much out of touch with “normal” people to realize just how irrationally many of them are likely to act.

  • Tim Tyler

    I’m not at present arguing over whether this hypothetical killing is good or bad. I just want people to realize that it is in the range of possible actions, and to own up to the fact that we are talking about a “the ends justifies the means” situation that would make Henry Kissinger flinch, rather than trusting that Papa AI will make everything better for everybody because he comes with a sticker that says “Friendly Inside!”

    Don’t want the end to justify the means? Isn’t the solution simply to prioritise the means more highly? I mean, if you are concerned that the machine intelligence will murder people – and you don’t think any end justifies that – then ISTM that you can simply prioritise its conformance to legal constraints that prohibit killing.

    ISTM that Eliezer’s plan amounts to trying to build a machine mind that voters in a democracy would approve of their government building – if they were consulted about the issue. Frankly, I doubt western voters would approve of a non-killing constraint – if it came to dealing swiftly with a proposed Islamic unkaafir (anti-infidel) superintelligence.

  • luzr

    Eliezer:

    Fair enought, esp. point (1) 🙂

    Anyway, I have some faint feeling that (3) in fact describes “bussines as usual”, only with a new powerful agent. At least, we can hope that emerging AI will not destroy humanity abruptly, because it will need it for a long time… (ok, 72 hours minimum, that is).

    But… I still cannot stop thinking that any sociopathic behaviour is not optimal, so as long as AI is really strong, it is not in its best interest to ‘misbehave’. Still, this depends on what it tries to optimize, which in the end is the purpose of this blog, right?

  • Tyrrell McAllister

    Tyrell read me as saying a cliche which was almost the opposite of what I said. This is a known hazard in saying non-cliche things.

    You’re right, I did misread you. My apologies.

  • http://shagbark.livejournal.com Phil Goetz

    Don’t want the end to justify the means? Isn’t the solution simply to prioritise the means more highly? I mean, if you are concerned that the machine intelligence will murder people – and you don’t think any end justifies that – then ISTM that you can simply prioritise its conformance to legal constraints that prohibit killing.

    Tim, you are one of the sharpest commentators here on OB, so it distresses me that after I have stated what I am trying to do, been misinterpreted, restated it, been misinterpreted in the same way again, and then restated it again, even you still can’t believe that I actually mean what I said.

    I am NOT ARGUING FOR OR AGAINST drastic measures. Eliezer has never admitted that there is any potential for such drastic measures in his scenarios. I want people to realize that there is that potential, because it needs to enter into the conversation. Too many people are seduced by the word “friendly” in “Friendly AI” into thinking that it means the AI will be their friend. But the way Eliezer uses the term “friendly” isn’t even analogous to its true meaning.

  • http://occludedsun.wordpress.com Caledonian

    Mr. Goetz is quite right.

    No matter how thoroughly we vet the ‘Friendliness’ of the AI, it is quite likely that the entity would eventually recommend some course of action that a person holding the plug would find unacceptable; said person would probably conclude the AI wasn’t really Friendly, and destroy it.

    If someone managed to build an intelligence, and the intelligence concluded that six billion people needed to be killed within the next decade to create the least-worst outcome it could sense, do we really think people would accept this?

    If the AI concluded that the past few thousand years of change in humanity’s moral systems were in error, do we really think people would reject their standards? Or would they insist the AI was in error because it rejected their standards and try to ‘fix’ it to reach the ‘correct’ results?

  • Tim Tyler

    Phil, thanks! Maybe it’s just that our views are usually pretty similar…

    You argued that “given an Eliezer-made AI, it is likely to kill people”.

    I am not sure about that. Eliezer’s proposed goal is a complicated extrapolation that neither I nor anyone else understands. Since the whole concept is pretty vague and amorphous, it seems rather difficult to say what it would actually do. Maybe it would kill people. However, you seemed to be claiming that it would be very likely to kill people.

    Now, maybe you have a better understanding of Eliezer’s proposal than I do. However, the way the whole thing is formulated suggests you would have to be a superintelligent agent to actually make much sense of it. That makes it difficult for critics such as yourself to get much of a fix on the target.

  • Spambot

    “I don’t think I can say it any more clearly: I’m not at present arguing over whether this hypothetical killing is good or bad.”
    I knew that, and agree that a CEV AI might kill quite a lot of people (in general I agree with Caledonian’s comment above), perhaps depending quite sensitively on the aggregation and extrapolation rules. I explained why I think it’s not very valuable to wrangle about the specifics, and why I was uninterested in Eliezer doing so and didn’t mind that he hasn’t spent a huge amount of time shocking people about it. Blunt statements of shocking conclusions are not that productive when they turn people off from considering the reasoning and general logic.

  • http://shagbark.livejournal.com Phil Goetz

    Spambot wrote:

    Blunt statements of shocking conclusions are not that productive when they turn people off from considering the reasoning and general logic.

    So you think that Eliezer avoids answering certain questions, because a little answer is a dangerous thing?

    I don’t think that explains his behavior well. And if it did, it would mean that he views us all as children, incapable of understanding the finer points, let alone actually contributing something.

    Tim Tyler wrote:

    I am not sure about that. Eliezer’s proposed goal is a complicated extrapolation that neither I nor anyone else understands. Since the whole concept is pretty vague and amorphous, it seems rather difficult to say what it would actually do. Maybe it would kill people. However, you seemed to be claiming that it would be very likely to kill people.

    I think that the possibility that it may kill people should be acknowledged. I think that in takeoff scenarios of a middling speed it is more likely to kill people; and that Eliezer and others have prematurely assigned all such scenarios a probability of zero.

    My motive in pointing this out is not to say that it may kill people and this would be bad. My motive is more along the lines of prodding people out of thinking “if we can make friendly AI then we will be saved”.

    My larger objective would be to point out that “we will be saved” is ill-defined; and that “saving humanity” will likely end up meaning something entailing the physical death of most humans, saving humans in a zoo with technological development forbidden, or something that we morally ought not to do.

    The presentation of CEV is schizophrenic because of this. On one hand, it’s supposed to save human values by extrapolating from them. On the other hand, Eliezer thinks that values are arbitrary; and whenever he talks about CEV, he talks about saving the future for meatly humans – as if the purpose of guiding the AI’s development were not for its effects on the AI, but for its benefits for meat-humans. I don’t know if this is confusion on his part, or a deliberate schizophrenia cultivated to avoid scaring off donors. Repeated questioning by me has failed to produce any statement from him to the effect that he imagines the humans he wants to save ever being anything other than they are today.

    Now, maybe you have a better understanding of Eliezer’s proposal than I do. However, the way the whole thing is formulated suggests you would have to be a superintelligent agent to actually make much sense of it. That makes it difficult for critics such as yourself to get much of a fix on the target.

    Eliezer is not a superintelligent agent. So your statement necessarily implies that CEV is nonsense.

    I would have a much better understanding of Eliezer’s proposal if he were willing to spend one-one-hundredth as much time answering simple questions about it, as he does writing about it.

    But I also think I am done with it. I have wasted too much time already trying to bring about a review of someone else’s ideas, when that person isn’t even interested in having his ideas reviewed.

    When people talk about the scientific method, they usually focus on the up-front part – making predictions and testing them. But another part of the scientific method is peer review. I can see how this would present problems for someone who imagines he has no peers. But “take it or leave it” is not operating within the scientific method.

  • Spambot

    “So you think that Eliezer avoids answering certain questions, because a little answer is a dangerous thing?

    I don’t think that explains his behavior well. And if it did, it would mean that he views us all as children, incapable of understanding the finer points, let alone actually contributing something.”

    OB is broadcast to everyone?

  • Pingback: Overcoming Bias : Emulations Go Foom

  • Pingback: What If? | διά πέντε / dia pente