Foom Debate, Again

My ex-co-blogger Eliezer Yudkowsky last June:

I worry about conversations that go into “But X is like Y, which does Z, so X should do reinterpreted-Z”. Usually, in my experience, that goes into what I call “reference class tennis” or “I’m taking my reference class and going home”. The trouble is that there’s an unlimited number of possible analogies and reference classes, and everyone has a different one. I was just browsing old LW posts today (to find a URL of a quick summary of why group-selection arguments don’t work in mammals) and ran across a quotation from Perry Metzger to the effect that so long as the laws of physics apply, there will always be evolution, hence nature red in tooth and claw will continue into the future – to him, the obvious analogy for the advent of AI was “nature red in tooth and claw”, and people who see things this way tend to want to cling to that analogy even if you delve into some basic evolutionary biology with math to show how much it isn’t like intelligent design. For Robin Hanson, the one true analogy is to the industrial revolution and farming revolutions, meaning that there will be lots of AIs in a highly competitive economic situation with standards of living tending toward the bare minimum, and this is so absolutely inevitable and consonant with The Way Things Should Be as to not be worth fighting at all. That’s his one true analogy and I’ve never been able to persuade him otherwise. For Kurzweil, the fact that many different things proceed at a Moore’s Law rate to the benefit of humanity means that all these things are destined to continue and converge into the future, also to the benefit of humanity. For him, “things that go by Moore’s Law” is his favorite reference class.

I can have a back-and-forth conversation with Nick Bostrom, who looks much more favorably on Oracle AI in general than I do, because we’re not playing reference class tennis with “But surely that will be just like all the previous X-in-my-favorite-reference-class”, nor saying, “But surely this is the inevitable trend of technology”; instead we lay out particular, “Suppose we do this?” and try to discuss how it will work, not with any added language about how surely anyone will do it that way, or how it’s got to be like Z because all previous Y were like Z, etcetera. (more)

When we shared this blog, Eliezer and I had a long debate here on his “AI foom” claims. Later, we debated in person once. (See also slides 34,35 of this 3yr-old talk.) I don’t accept the above as characterizing my position well. I’ve written up a summaries before, but let me try again, this time trying to more directly address the above critique.

Eliezer basically claims that the ability of an AI to change its own mental architecture is such a potent advantage as to make it likely that a cheap unnoticed and initially low ability AI (a mere “small project machine in a basement”) could without warning over a short time (e.g., a weekend) become so powerful as to be able to take over the world.

As this would be a sudden big sustainable increase in the overall growth rate in the broad capacity of the world economy, I do find it useful to compare to compare this hypothesized future event to the other pasts events that produce similar outcomes, namely a big sudden sustainable global broad capacity rate increase. The last three were the transitions to humans, farming, and industry.

I don’t claim there is some hidden natural law requiring such events to have the same causal factors or structure, or to appear at particular times. But I do think these events suggest a useful if weak data-driven prior on the kinds of factors likely to induce such events, on the rate at which they occur, and on their accompanying inequality in gains. In particular, they tell us that such events are very rare, that over the last three events gains have been spread increasingly equally, and that these three events seem mainly due to better ways to share innovations.

Eliezer sees the essence of his scenario as being a change in the “basic” architecture of the world’s best optimization process, and he sees the main prior examples of this as the origin of natural selection and the arrival of humans. He also sees his scenario as differing enough from the other studied growth scenarios as to make analogies to them of little use.

However, since most global bio or econ growth processes can be thought of as optimization processes, this comes down to his judgement on what counts as a “basic” structure change, and on how different such scenarios are from other scenarios. And in my judgement the right place to get and hone our intuitions about such things is our academic literature on global growth processes.

Economists have a big literature on processes by which large economies grow, increasing our overall capacities to achieve all the things we value. There are of course many other growth literatures, and some of these deal in growths of capacities, but these usually deal with far more limited systems. Of these many growth literatures it is the economic growth literature that is closest to dealing with the broad capability growth posited in a fast growing AI scenario.

It is this rich literature that seems to me the right place to find and hone our categories for thinking about growing broadly capable systems. One should review many formal theoretical models, and many less formal applications of such models to particular empirical contexts, collecting “data” points of what is thought to increase or decrease growth of what in what contexts, and collecting useful categories for organizing such data points.

With such useful categories in hand one can then go into a new scenario such as AI foom and have a reasonable basis for saying how similar that new scenario seems to old scenarios, which old scenarios it seems most like if any, and which parts of that new scenario are central vs. peripheral. Yes of course if this new area became mature it could also influence how we think about other scenarios.

But until we actually see substantial AI self-growth, most of the conceptual influence should go the other way. Relying instead primarily on newly made up categories and similarity maps between them, concepts and maps which have not been vetted or honed in dealing with real problems, seems to me a mistake. Yes of course a new problem may require one to introduce some new concepts to describe it, but that is hardly the same as largely ignoring old concepts.

So, I fully grant that the ability of AIs to intentionally change mind designs would be a new factor in the world, and it could make a difference for AI ability to self-improve. But while the history of growth over the last few million years has seen many dozens of factors come and go, or increase and decrease in importance, it has only seen three events in which overall growth rates greatly increased suddenly and sustainably. So the mere addition of one more factor seems unlikely to generate foom, unless our relevant categories for growth causing factors suggest that this factor is unusually likely to have such an effect.

This is the sense in which I long ago warned against over-reliance on “unvetted” abstractions. I wasn’t at all trying to claim there is one true analogy and all others are false. Instead, I argue for preferring to rely on abstractions, including categories and similarity maps, that have been found useful by a substantial intellectual community working on related problems. On the subject of an AI growth foom, most of those abstractions should come from the field of economic growth.

GD Star Rating
loading...
Tagged as: , , , ,
Trackback URL:
  • Rafal Smigrodzki

    One possible difference between usual forms of economic growth and the AI explosion scenario is that economic growth is generated by independent decision-makers interacting in both cooperative and competitive ways, while the AI might be a single decision maker – therefore no longer acting under the constraints imposed by competition among relatively similar entities (which in turn has strong analogies to evolution). The AI singleton, perhaps much more intelligent than us, could copy itself in large numbers, thus generating the huge labor force needed to effectively outcompete others, yet without necessarily triggering competition among the copies – these could have a stable goal system compatible with parallel existence. Such a distributed goal system would be a first in the history intelligent beings, potentially impacting us as much as the development of eusociality affected the mass-balance among insects.

    • http://richardloosemore.com/ Rloosemore

       I concur.  If we put to one side arguments about whether such a singleton is likely to emerge, and only ask what would happen if it did emerge, it becomes clear that a Singleton-With-Coherent-Purpose scenario is pretty much the opposite of evolutionary or economic models, so the latter models could be rendered obsolete in their entirety.

      • RLoosemore

         This is a test comment to find out whether Disqus is going crazy…

    • http://kruel.co/ Alexander Kruel

      …the AI might be a single decision maker – therefore no longer acting under the constraints imposed by competition among relatively similar entities…

      Constraints? Culture and competition is a key advantage. Would a Babylonian mathematician discover modern science and physics given that he would be uploaded into a supercomputer (a possibility that is in and of itself already highly speculative)? It assumes that he could brute-force conceptual revolutions.Even if he was given a detailed explanation of how his mind works and the resources to understand it, self-improving to achieve superhuman intelligence assumes that throwing resources at the problem of intelligence will magically allow him to pull improved algorithms from solution space as if they were signposted.If people like Benoît B. Mandelbrot would have never decided to research Fractals then many modern movies wouldn’t be possible, as they rely on fractal landscape algorithms. Yet, at the time Benoît B. Mandelbrot conducted his research it was not foreseeable that his work would have any real-world applications.Important discoveries are made because many routes with low or no expected utility are explored at the same time. And to do so efficiently it takes random mutation, a whole society of minds, a lot of feedback and empirical experimentation.“Treating rare diseases in cute kittens” might or might not provide genuine insights and open up new avenues for further research. As long as you don’t try it you won’t know.The idea that a rigid consequentialist with simple values can think up insights and conceptual revolutions simply because it is instrumentally useful to do so is implausible.Complex values are the cornerstone of diversity, which in turn enables creativity and drives the exploration of various conflicting routes. A singleton with a stable utility-function lacks the feedback provided by a society of minds and its cultural evolution.You need to have various different agents with different utility-functions around to get the necessary diversity that can give rise to enough selection pressure.More here.

      • Margin

        Enough selection pressure for what?

        All a singleton needs is the means to irreversibly dominate all humans.

        Or kill them.

      • Margin

        From your website:

        “Imagine a group of 100 world-renowned scientists and military strategists. The group is analogous to the initial resources of an AI. The knowledge that the group has is analogous to what an AI could
        come up with by simply “thinking” about it given its current resources. Could such a group easily wipe away the Roman empire when beamed back in time?”

        Yes.

        If it really wanted to, the group could destroy the Roman empire.

        If you think otherwise, that’s because you are not 100 world-renowned scientists and strategists.

        I would use bio weapons.

      • http://kruel.co/ Alexander Kruel

        Margin, the problem is that you can’t magically create bioweapons just by reasoning about them. Intelligence alone is not a sufficient feature. 

        Consider that we are already at a point where we have to build billion dollar chip manufacturing facilities to run our mobile phones. We need to build huge particle accelerators to obtain new insights into the nature of reality. It takes a whole technological civilization to produce a modern smartphone.

        And even if an AI was able to somehow acquire large amounts of money. It is not easy to use the money. You can’t “just” build huge companies with fake identities, or a straw man, to create revolutionary technologies easily. Running companies with real people takes a lot of real-world knowledge, interactions and feedback. But most importantly, it takes a lot of time. An AI could not simply create a new Intel or Apple over a few years without its creators noticing anything.

        All a singleton needs is the means to irreversibly dominate all humans.

        Easier said than done! To quote Holden Karnofsky from GiveWell,

        Humans are enormously more intelligent than mosquitoes; humans are good at predicting, manipulating, and destroying mosquitoes; humans do not value mosquitoes’ welfare; humans have other goals that mosquitoes interfere with; humans would like to see mosquitoes eradicated at least from certain parts of the planet. Yet humans haven’t accomplished such eradication…

        Can you answer the following questions and taboo “intelligence” in doing so?

        1.) How is an AI going to become a master of dark arts and social engineering in order to persuade and deceive humans?

        2.) How is an AI going to coordinate a large scale conspiracy or deception, given its initial resources, without making any suspicious mistakes along the way?

        3.) How is an AI going to hack the Internet to acquire more computational resources?Are those computational resources that can be hacked applicable to improve the general intelligence of an AI?

        4.) Does throwing more computational resources at important problems, like building new and better computational substrates, allow an AI to come up with better architectures so much faster as to outweigh the expenditure of obtaining those resources, without hitting diminishing returns?5.) Does an increase in intelligence vastly outweigh its computational cost and the expenditure of time needed to discover it?6.) How can small improvements replace conceptual revolutions that require the discovery of unknown unknowns?
        7.) How does an AI brute-force the discovery of unknown unknowns?
        8.) Is an agent of a given level of intelligence capable of handling its own complexity efficiently?
        9.) How is an AI going to predict how improvements, respectively improved versions of itself, are going to act, to ensure that its values are preserved?10.) How is an AI going to solve important problems without real-world experimentation and slow environmental feedback?
        11.) How is an AI going to build new computational substrates and obtain control of those resources without making use of existing infrastructure?12.) How is an AI going to cloak its actions, i.e. its energy consumption etc.?
        13.) How is an AI going to stop humans from using its own analytic and predictive algorithms in the form of expert systems to analyze and predict its malicious intentions?14.) How is an AI going to protect itself from human counter strikes given the fragility of the modern world and its infrastructure, e.g. without some sort of shellproof internal power supply?

      • Margin

        You don’t need magic for bio weapons.

        They are real.

        Applied intelligence was enough to make them real.

        Now that the knowledge exists, applied intelligence can use it to create more effective ones faster.

        There have been animal tests of bio agents that kill 100% of all hosts in a population.

      • http://kruel.co/ Alexander Kruel

        You don’t need magic for bio weapons.

        The magic is in your implicit and unwarranted assumptions:

        1.a.) You assume a superintelligence could somehow obtain access to the necessary hardware and resources to produce bioweapons.

        1.b.) You assume that #1.a. could happen in such a way that humans don’t notice it.

        2.) You assume that a superintelligence could somehow acquire the means to deploy those bioweapons and kill a sufficient number of humans quickly enough to prevent them from any counter-strike.

        3.) You assume that such a superintelligence does not depend on humans and their industrial civilization.

        4.) You assume that a superintelligence would bother to kill humans at all.

        …and you make countless other assumptions about how such a superintelligence would be developed, how quick it would happen and how it would never show any malicious intentions during its development process and only start trying to kill humans after it became powerful enough.

        I could go on for days listing assumptions you make without having any evidence that they are at all warranted.

      • Margin

        1a) Yes, I think this is a plausible assumption.

        1b) Yes, or that the relevant humans are unwilling or unable to prevent it.

        2) Yes, or prepare for a counter-strike in such a way that the AI can survive it with a sufficient power base.

        3) Yes, I think this is a plausible assumption, if you take “depend on” as “cannot survive with a power base when all humans die”.

        4) This is a very highly plausible assumption.

        If I were 1000 times smarter, could copy myself without error, distribute myself over the globe this way and lived in a substrate that can survive bio weapons, I would try to kill all humans.

        A 0.1% shot at world domination would be worth any risk of retaliation.

      • http://kruel.co/ Alexander Kruel

        If I were 1000 times smarter…

        You have absolutely no clue what that means. You just create a string of English words and make unwarranted inferences from those words. Which makes it really hard to take your confidence seriously.

        2) Yes, or prepare for a counter-strike in such a way that the AI can survive it with a sufficient power base.

        Ridiculous. How would it prepare for that? Does it order a bunker at eBay or what?

        …could copy myself without error, distribute myself over the globe this way and lived in a substrate that can survive bio weapons, I would try to kill all humans.

        The first part is complete hand waving. You have no idea how to do that and just use intelligence as a fully general counterargument. The second part is pure anthropomorphization of what an artificial intelligence would want to do based on what you would do.

      • Margin

        “You have absolutely no clue what that means. You just create a string of English words and make unwarranted inferences from those words.”

        You do realize that these sentences could be copied and pasted into any discussion about any topic?

        I suggest you learn to discuss interesting topics without becoming needlessly emotional.

        Then you don’t have to fall back on such rhetoric.

        “Ridiculous. How would it prepare for that? Does it order a bunker at eBay or what?”

        I would use redundancy over local defense. This is why the internet was created in the first place.

        “The first part is complete hand waving. You have no idea how to do that
        and just use intelligence as a fully general counterargument.”

        We have no idea how to copy digital data without error and distribute it globally?

        Are you serious?

        “The second part is pure anthropomorphization of what an artificial intelligence would want to do based on what you would do.”

        Killing off all competition with different goals is an obvious choice for any agent who wants to maximize something specific.

      • http://kruel.co/ Alexander Kruel

        You do realize that these sentences could be copied and pasted into any discussion about any topic?

        Now you are just evading the fact that you have no idea what “1000 time smarter” means yet make complex inferences based on it.

        It is very often useful to use certain natural language terminology to ease communication between humans who share a common and well-defined understanding of those terms. Which is not the case here.

        Then you don’t have to fall back on such rhetoric.

        If you don’t outline what you mean by “such rhetoric” I have to assume that you are falling back on attacking me because you have no arguments left to defend your claims.

        I would use redundancy over local defense. This is why the internet was created in the first place.

        The global infrastructure is highly depended on humans. If you additionally turn those humans against you by trying to kill them then it becomes effectively impossible to keep everything running in such a way as to allow you to take over the world.

        We have no idea how to copy digital data without error and distribute it globally?

        We’re talking about the first artificial general intelligence here not some PDF file. It is highly suspicious if it would be easily copy-able without anyone noticing it and could additionally make use of other resources in such a way as to retain its capabilities. Even given that we disregard improved security measures that will very likely be in place in future.

        Killing off all competition with different goals is an obvious choice for any agent who wants to maximize something specific.

        Even if I was to grant that you had any idea how such an intelligence was going to be developed and that a consequentialist AI was at all possible I would say that such a drive is very unlikely.

        Not only would such an intelligence have to show no signs of trying to kill all competition during its developmental stage, because otherwise it would be noticed, but it would also have to be designed with the goal of unbounded maximization and a definition of “maximization” that is sufficiently detailed to allow it to draw action relevant conclusions such as “killing all humans”.

      • Margin

        “Now you are just evading the fact that you have no idea what “1000 time smarter” means yet make complex inferences based on it.”

        Let’s say if I had eidetic memory, considerably expanded working memory space and could think for 1000 current subjective second in each actually passing second, I would consider myself “1000 times smarter”.

        This does not include enhanced cognitive algorithms compared to human reasoning, even though there are probably some that could be discovered.

        “If you don’t outline what you mean by “such rhetoric”"

        I meant the “you have no idea, you just use english words and make up inferences” part. Never mind.

        “The global infrastructure is highly depended on humans. If you
        additionally turn those humans against you by trying to kill them then
        it becomes effectively impossible to keep everything running in such a
        way as to allow you to take over the world.”

        That is all correct, but I would replace “effectively impossible” with “very hard”.

        “We’re talking about the first artificial general intelligence here not some PDF file.”

        This doesn’t really matter.

        Encryption allows us to share functional data that no government wants us to have.

        Some of us do it all the time.

        “Not only would such an intelligence have to show no signs of trying to
        kill all competition during its developmental stage, because otherwise
        it would be noticed,”

        Showing that you want to kill everyone is stupid when you don’t have the power to pull it off – unless the other side can predict you will never have the power.

        This is why I can admit it (you know I’m practically powerless), but an functional AI that understands human psychology will be kiss-ass friendly until the point when switching strategies becomes worth it.

        “but it would also have to be designed with the
        goal of unbounded maximization and a definition of “maximization” that
        is sufficiently detailed to allow it to draw action relevant
        conclusions such as “killing all humans”.”

        I think this is plausible enough within the design space relevant for AIs developed by corporations and the military to fulfill real-world goals.

        Three things to add:

        1) I do not think sudden foom is high probability. It is low-probability but high-stakes.

        2) I agree ‘foom’ doesn’t have to happen in one weekend to be high-stakes, it could take 30 years and would still be high-stakes.

        3) You mention improved security in the future. I would turn this around and focus on increased total computing power that is networked and accesses more total hardware power in the future.

      • http://kruel.co/ Alexander Kruel

        Let’s say if I had eidetic memory, considerably expanded working memory space and could think for 1000 current subjective second in each actually passing second, I would consider myself “1000 times smarter”.

        So what is the net advantage of eidetic memory if you consider that humans can use tools to effectively achieve the same?

        And with respect to a larger working memory. What advantage is there between humans who can extent their working memory using their tools and an AI? Just because we make a certain kind of psychological distinction between things we can hold in our mind without tools, and things we can’t, does not mean there is some radical qualitative advantage (as opposed to the obvious speed advantages) in increasing the capacity of working memory.

        If an AI that we invented can hold a complex model in its mind, then we can also simulate such a model by making use of expert systems. Being consciously aware of the model doesn’t make any great difference in principle to what you can do with the model.

        The same goes for having more thoughts in a shorter time. A collective of humans and their tools can also think much faster than a single human being. Yet the advantage isn’t that great. Sometimes a single human being can outsmart humanity. Yet humanity can kill a single human being. Which indicates that it is far from being clear that greater intelligence equals faster thinking or greater power.

        For more serial power to give an decisive advantage, important problems would have to fall into complexity classes where throwing more computational resources at a problem does not lead to diminishing returns.

        Similar for parallel computation. It is not clear how many tasks are easily decomposable into smaller operations.

        The questionable assumptions here are 1.) that throwing additional computational resources at important problems solves them proportionally faster, fast enough to overpower humans who are in control of the planets infrastructure and resources 2.) that important problems are parallelizable.

        Consider that the U.S. has many more and smarter people than the Taliban. The bottom line is that the U.S. devotes a lot more output per man-hour to defeat a completely inferior enemy. Yet they are losing.

        The problem is that you won’t beat a human at Tic-tac-toe just because you thought about it for a million years.

        You also won’t get a practical advantage by throwing more computational resources at the travelling salesman problem and other problems in the same class.

        You are also not going to improve a conversation in your favor by improving each sentence for thousands of years. You will shortly hit diminishing returns. Especially since you lack the data to predict human opponents accurately.

        This does not include enhanced cognitive algorithms compared to human reasoning, even though there are probably some that could be discovered.

        Is it possible to quantify those advantages in such a way to conclude that they outweigh all the cognitive tricks that have been accumulated over billions of years of evolution?

        I haven’t seen any evidence that most evolutionary designs are vastly less efficient than their technological counterparts. A lot of the apparent advantages of technological designs is a result of making wrong comparisons like between birds and rockets. We haven’t been able to design anything that is nearly as efficient as natural flight. It is true that artificial flight can overall carry more weight. But just because a train full of hard disk drives has more bandwidth than your internet connection does not imply that someone with trains full of HDD’s would be superior at data transfer.

        Encryption allows us to share functional data that no government wants us to have.

        But we’re talking about a state of the art technology which will likely run on state of the art hardware, not one AI of a huge ecosystem of different AI’s that run everywhere from smartphones to personal computers.

        To imagine that an AI could simply copy itself would be similar to imaging that IBM’s Blue Brain Project could simply be copied in such a way that not only nobody notices the unexpected use of bandwidth and surge up of everyone’s CPU load but that it would run effectively enough to make it worthwhile to take the risk of detection and increased instability due to using highly volatile infrastructure that was never adapted to run such a software.

        Showing that you want to kill everyone is stupid when you don’t have the power to pull it off – unless the other side can predict you will never have the power.

        Realize that it would have to be able to hide such intentions even during the developmental phase when it was not yet able to improve itself and learn how to deceive humans. Because otherwise its developers would either abandon the project or make sure that those intentions are eliminated. And it seems very unlikely that it could manage to do so.

      • http://kruel.co/ Alexander Kruel

        1) I do not think sudden foom is high probability. It is low-probability but high-stakes.

        Where do you draw the line though?

        I don’t think that reason for mitigating AI risks are workable. And even if it was, you are not pursuing them consistently:

        1.a.) It is never possible to completely rule out long-term detriments. Any action can have negative expected utility if only you conjecture a sufficient loss.

        You can just wipe off everything with expected utility maximization. In theory you could even argue that criticizing MIRI (formerly known as the Singularity Institute) at all has negative expected utility because if they are right you will ever so slightly reduce the chance of a positive Singularity and if they are wrong they will just waste a bit more money which would probably be wasted anyway.

        Almost any activity has negative expected utility if you account for the possibility that you might die pursuing it, just by taking into account the fun you might have living for billions of years in an inter-galactic civilization.

        1.b.) It is always possible to conjecture a scenario with lower probability yet sufficient expected utility to outweigh all other scenarios.

        2.) If your argument is that we should not discount AI risks because of the expected utility involved then you have to explain how you discount scenarios that bear similarity to Pascal’s mugging.

        3.) If your argument is that we should discount low probability scenarios, even though their value is sufficiently larger to outweigh their reduced likelihood, then you have to explain how such an objection is not always possible, e.g. in case of AI risks.

        4.) If your argument is that AI risk is the most probable underfunded risk then what is the greatest lower bound for “probable” here and how do you formally define it? In other words, “probable” in conjunction with “underfunded” doesn’t work either because any case of Pascal’s mugging is underfunded as well. You’d have to formally define and justify some well-grounded minimum for “probable”.

        Conclusion:

        If you argue that it is more reasonable to contribute to the mitigation of risks associated with artificial general intelligence than to contribute to more or less probable risks then, in case you are not just appealing to intuition, there must be some formalized argument that favors AI risk mitigation over all other possible actions.  In other words, you need to formally define “reasonable”.

        The only way to argue in favor of AI risks is to evade the well-established heuristic that extraordinary claims require extraordinary evidence. Which leads to all kinds of problems, as partly outlined above.

        The line of reasoning adopted by AI risks advocates is simply unworkable for computationally bounded agents like us. We are forced to arbitrarily discount certain obscure low probability risks or else fall prey to our own shortcomings and inability to discern fantasy from reality. In other words, it is much more probable that we’re going make everything worse or waste our time than that we’re actually maximizing expected utility when trying to act on such vague ideas as an intelligence explosion.

      • Margin

        “If you argue that it is more reasonable to contribute to the mitigation of risks associated with artificial general intelligence than to…”

        Whoa, whoa, wait a second.

        I never claimed any of that.

        I wouldn’t spend any money to affect what happens after I die or after I’m already old.

        I just critized your assumption that foom is implausible.

        It is hard but worth for some agent who can pull it off with probability > 0.1%

        And I think given AGI it’s at least 0.1%

      • http://kruel.co/ Alexander Kruel

        I just critized your assumption that foom is implausible.

        I had to split the comment because it got too long. See my other comment below the one you commented on.

      • Margin

        “The same goes for having more thoughts in a shorter time. A collective
        of humans and their tools can also think much faster than a single
        human being.”

        Uh, no, they can’t.

        Instead they claw at each other like harpies.

        Even if you look at a “success” such as wikipedia, half of the pages are locked because people have different ideologies and can’t commit to NPOV.

        Most people hate each other fiercly, but I don’t hate myself, and neither will an AGI.

        If I could copy myself without error, I’d know exactly what the other copies want and I would trust them implicitly.

        And you would never see us coming.

      • VV

         

        If I could copy myself without error, I’d know exactly what the other copies want and I would trust them implicitly.

        And you would be misplacing your trust, since once your copies have diverged a little bit, which is inevitable since they would have different experiences, you could no longer predict that their behavior would be the same as yours.

      • Margin

        “And you would be misplacing your trust, since once your copies have
        diverged a little bit, which is inevitable since they would have
        different experiences, you could no longer predict that their behavior
        would be the same as yours.”

        Yeah, that sounds smart.

        But it’s no more than a technicality compared to the conflicts other people push into each other’s lives.

        Also an AGI would probably not change their terminal values because of experience, which is what humans do because they’re incoherent meat-bulbs and which is totally irrational from any coherent agency POV

      • VV

         @2f38900f331b0065d7d9cb4653aff0ec:disqus

        Let’s say if I had eidetic memory, considerably expanded working memory
        space and could think for 1000 current subjective second in each
        actually passing second, I would consider myself “1000 times smarter”.

        It seems that you are confusing implementation with function. Memory space and thinking speed are things which are required to have intelligence, but they are not themselves measures of intelligence. The ability to solve problems is a measure of intelligence.

        If your brain was made 1000 times faster and had more working memory, would you be better at problem solving? Probably. Would you 1000 times better? Not a chance. You would probably not even 100 times better.

      • VV

        @2f38900f331b0065d7d9cb4653aff0ec:disqus 

        But it’s no more than a technicality compared to the conflicts other people push into each other’s lives.

        No.

        The “Prisoner’s Dilemma with Mental Clone” gedankenexperiment, where cooperation is the optimal choice, works on a technicality, namely that since you can predict that the other player will make (with high probability) the same choice as yours, then the only possible outcomes are that you both cooperate or you both defect, and the former is preferable to the latter.

        But in any realistic scenario, you can’t make the assumption that the other player will behave like you, even if at some point in the past you were copies.

        You might try to invoke cooperation based on some argument along the lines of superrationality. But superrationality stands on much more shaky ground than standard game theory: it doesn’t seem to explain the behavior of humans (who in order to cooperate, often need enforceable laws and contracts, or deeply ingrained moral codes). Why should you expect it to apply to non-equal copies of the same agent?

        Keep in mind that other people are already almost perfect copies of yourself, given the overall genetic similarity between all humanity, and also the cultural similarity between you and those you normally interact with. Just compare human behaviors to the wide range of behaviors observed in animals. Yet, even with all that similarity, cooperation between fellow humans is far from granted.

        Also an AGI would probably not change their terminal values because of
        experience, which is what humans do because they’re incoherent
        meat-bulbs and which is totally irrational from any coherent agency POV

        1. Why should you expect AGI to be much more rational than humans? Humans are only as rational as evolution made them. An AGI would be only as rational as its maker made it.

        2. You don’t need to change terminal values to achieve a failure to cooperate. In Prisoner’s Dilemma, the two players have the same payoff matrix, that is, the same terminal values. Nevertheless, unless the two players are equal, they will not cooperate.

      • Margin

        Now that you started “liking” each other like faggots, the whole comment order is screwed up.

        Good job lol

      • http://kruel.co/ Alexander Kruel

        You can sort by oldest first.

      • Margin

        “You can sort by oldest first.”

        Oh, I hadn’t seen that.

        This is useful.

      • Margin

        @5dcdf28d944831f2fb87d48b81500c66:disqus

        “But in any realistic scenario, you can’t make the assumption that
        the other player will behave like you, even if at some point in the
        past you were copies.”

        I know they would share my values, unless there is too much change in between.

        I know that I don’t change so much over short periods of time.

        I would cooperate with copies of myself, so unless they were brainwashed, they would do the same.

        “Keep in mind that other people are already almost perfect copies of
        yourself, given the overall genetic similarity between all humanity,
        and also the cultural similarity between you and those you normally
        interact with.”

        Lol no.

        Most people care about completely differnt things than I do.

        “1. Why should you expect AGI to be much more rational than humans?
        Humans are only as rational as evolution made them. An AGI would be
        only as rational as its maker made it.”

        Intelligent design > evolution.

        This will be the first time intelligent design makes minds that can reason.

        “2. You don’t need to change terminal values to achieve a failure to
        cooperate. In Prisoner’s Dilemma, the two players have the same payoff
        matrix, that is, the same terminal values. Nevertheless, unless the two
        players are equal, they will not cooperate.”

        Payoff matrix != terminal values.

        A terminal value is an expression how you want the world to be structured.

        I want the world to be structured differently than most people do, but identical to how copies of me would.

        That makes cooperating instead of defecting rational, and a functional (not insane) AI would do the same.

    • Robin Hanson

      The effect of having a single decision maker, well-coordinated internally, on growth rates is the sort of factor that current social science is well placed to analyze. If you want good intuitions about how much such a factor could influence growth rates, you would do well to learn and apply current social science. This is *not* an unprecedented neglected factor that invalidates all attempts to compare this new situation to previously studied situations. 

      • Rafal Smigrodzki

        Current social science is a large endeavor devoted to analyzing interactions between multiple, evolved and competing decision makers with incongruent goal systems, so your pointer is very broad. Can you give references to specific publications which could help understand a designed, multi-copy, single-goal-system superhuman AI?
        Surely, the peculiar features of likely AI do not invalidate all our current analytic approaches but they could make it difficult to use them.

  • Tim Tyler

    Have your proposed three previous growth modes been corroborated by other students of the subject?

    It seems to me that most other folk model the past more continuously – using exponential growth and gradualistic models involving increasing synergetic interactions.
     

  • Rloosemore

    Robin, you seem to accept that none of the comparison events you cite involved a factor that was itself intelligent, or capable of direct self-redesign.  (So:  Farms are not intelligent; industry is not itself intelligent; and neither of them were (ipso facto) capable of self-redesign).

    But then you proceed to do something extraordinary.  You want to justify your belief that economics should describe this new regime, so you execute the most amazing hand-wave:  “Yes of course a new problem may require one to introduce some new
    concepts to describe it, but that is hardly the same as largely ignoring
    old concepts.”

    The change from Descartes’ vortices to Newton’s universal theory of gravitation required us to ignore old concepts by the bucketload (ask Decartes).  The change from phlogiston to ideas about oxidation, ditto.  The change from caloric fluid to kinetic theory of heat.  Relativitistic mechanics required us to discard the ether and the concept of absolute frames of reference and simultaneity.  The change from pre-quantum to post-quantum physics required …… and on and on.

    And none of those changes involved new factors that were both intelligent and capable of self-redesign!  None.  So all of those previous changes were trivial in their capacity to disrupt old ideas, compared with the introduction of self-modifying AI.

    You are sounding like a Victorian aristocrat singing “Rule Britannia!, Britannia Rules The Waves! …” just before the end of Empire.

    • Tim Tyler

      Corporations are intelligent and capable of self-redesign.  There are some parts of themselves that they can’t yet redesign – but that seems to be no big deal.  Thus the *current* explosion in technology.

      • VV

        What current explosion in technology? A new version of iPhone every year?

      • Tim Tyler

        E.g. see Moore’s law.  Such exponential growth is the same thing that happens inside nuclear bombs.

      • VV

         @google-a6523c96cdda9eea0afe282c76640b73:disqus Serial speed pretty much stopped increasing.

        And anyway, increases in raw processing power don’t translate to proportional increases in actual utility. Your brand new PC doesn’t improve your life twice than the PC you bought 18 months ago.

      • Tim Tyler

        The serial speed plateau is only because they keep trying to keep a larger and larger number of components in synchrony.  I said we were seeing a “technology explosion” – not a “utility explosion”.  We can’t really measure utilities on an absolute scale.

      • VV

         

        The serial speed plateau is only because they keep trying to keep a larger and larger number of components in synchrony.

        Reference?

        I said we were seeing a “technology explosion” – not a “utility
        explosion”.  We can’t really measure utilities on an absolute scale.

        But we can measure, or at least estimate, utilities on a relative, personal scale. Talking about a “technological explosion” without reference to utility seems rather vague.

    • VV

       Organizations are collective agents that can redesign themselves. In recent times, many organizations do that in a principled way, or at least, they try.

      Do they achieve intelligence explosion? Nope.

      • Fadeway

        By the same token, you can redesign yourself by changing your heuristics to a limited extent, yet there isn’t an intelligence explosion. How surprising.

        This is a matter of extent/scale. An AI can change everything about itself. You can change almost nothing about yourself. An organization can’t redesign most of itself either.

      • VV

        An AI can change everything about itself.

        Wrong. At any point, the changes it can make to itself are limited by its material resources (hardware) and its current cognitive abilities.

        And anyway, most AI problems are conjectured to have at least exponential complexity. Which means that even if you are perfectly optimized to solve them, they are still difficult.

    • http://kruel.co/ Alexander Kruel

      Note that “intelligent and capable of self-redesign” might appear to be relatively simple and appealing concepts. But most of this superficial simplicity is a result of the vagueness of natural language descriptions. Reducing the vagueness of those concepts by being more specific, or by coming up with technical definitions of each of the words they are made up of, reveals the hidden complexity that is comprised in the vagueness of the terms.

      Most of the alleged capabilities of a self-improving artificial general intelligence are a result of vagueness, due to the assumption of a magic black box. Which also means that people critical of those capabilities can hardly argue specifically about how it would fail other than by looking at previous growth causing factors.

  • http://www.facebook.com/rspowell7 Robert S. Powell

    I love the word “foom” for the coming explosion in machine intelligence.

  • Abram Demski

    Robin, thank you for keeping this conversation up– it is appreciated by singinst “outsiders” such as myself (and, I take it, Richard Loosemore).

    On this particular occasion, your post has only served to convince me of the accusation of Eliezer (which I was not previously aware of): you try to justify the important role your preferred reference class has to play, but fundamentally, you are still making the sort of bad argument which Eliezer was complaining about.

    To sum it up, you are making “downward” predictions from chosen reference classes to AI consequences. It seems much more powerful to make “upward” predictions from specific properties & scenarios to consequences. 

    Of course, both downward and upward reasoning are absolutely needed. (Simplisticly put, that is how Bayesian reasoning works.) The upward arguments seem to outweigh the sorts of downward arguments you are considering, though.

    So, for example, it seems more interesting to discuss how hard/easy it might be to make major improvements via self-modification.

    • Jess Riedel

      “downward/upward reasoning” = “outside/inside view”, right? This is clearly a significant sticking point for Hanson and Yudkowsky, as they have discussed this meta reasoning issue before.

      • Abram Demski

        Not exactly, but that is close to what I mean. I’m visualizing down vs up as in a Bayesian network (which is a form of prior vs likelihood, combining via bayes law to make a posterior), but more concretely, the distinction I’m making is reasoning about an instance based on a class vs reasoning about the instance based on its parts and how they fit together.

      • VV

         Beware of the choice of the arrows in a Bayesian network. Except special cases, the direction of the arrow is arbitrary to some extent.

    • http://kruel.co/ Alexander Kruel

      I reject that it is at all useful to make non-evidence-backed speculations on possible bad outcomes. We just don’t know enough about artificial general intelligence and how it is going to be developed to draw reliable conclusions.Take for example what Nick Bostrom said here:

      Because if you can build new systems, even if all it could initially do is this type engineering work, you can build yourself a poetry module or build yourself a social skills module, if you have that general ability to build.

      Really? How do you know? This is complete bullshit. A fantasy made up on the fly. Indeed the whole line of reasoning in favor of AI risks is mostly based on pure fantasy. E.g. that an AI could somehow become a master of social engineering by inferring social skills, accumulated over millions of years of evolution, from watching YouTube videos.

      Just imagine you emulated a grown up human mind and it wanted to become a pick up artist, how would it do that with an Internet connection? It would need some sort of avatar, at least, and then wait for the environment to provide a lot of feedback.

      Therefore even if we’re talking about the emulation of a grown up mind, it will be really hard to acquire some capabilities. Then how is the emulation of a human toddler going to acquire those skills? Even worse, how is some sort of abstract AGI going to do it that misses all of the hard coded capabilities of a human toddler?

      Can we even attempt to imagine what is wrong about a boxed emulation of a human toddler, that makes it unable to become a master of social engineering in a very short time?

      Can we imagine what is missing that would enable one of the existing expert systems to quickly evolve vastly superhuman capabilities in its narrow area of expertise? Why haven’t we seen a learning algorithm teaching itself chess intelligence starting with nothing but the rules?

      • http://www.facebook.com/CronoDAS Douglas Scheinberg

         > Why haven’t we seen a learning algorithm teaching itself chess intelligence starting with nothing but the rules?

        You haven’t looked?

    • Robin Hanson

      The argument I hear is “it sure seems like a system that could redesign its mind architecture could grow a lot faster than a system that cannot.” There *is* no more specific mechanical or causal detail being offered as a basis for this prediction than this intuition. So it all comes down to intuitions about what factors are how potent in increasing growth. I suggest one should hone such intuitions in the context of other closely related problems already being solved by a large academic community.

      • Abram Demski

        I think we are at a point where we can begin to asses this in a (somewhat!) less speculative way. AI has been in development for 50 years. This gives weak indication of the ways in which human-level intelligence can and cannot come up with better algorithms than the brain. We’ve also been studying automatic programming for a while now, so we have some basis for the discussion of self-improving systems. We have a good-enough-for-discussion theory of universal intelligence, and we can talk about the basic computational complexity of approximating that, and its properties once approximated. None of this is conclusive, but it seems closer to the topic than historical economics.

      • VV

         It seem that, to some extend, Hanson and Yudkowsky are talking past to each other. Let me try to figure out their positions:

        Yudkowsky: Self-improving AI will bring forth an enormously transformative change to life as we know it.

        Hanson: Given the historical record, the prior for any event causing such a disruptive change is very low, therefore you are probably wrong.

        Yudkowsky: But the event I’m talking about is not like the others! It has *self-improvement*! So you can’t put it in the same reference class of other events.

        Hanson: You didn’t provide any strong argument to support the notion that the property of *self-improvement* makes such a big difference in determining the magnitude of the event, thus, I’m sticking to the prior.

      • Robin Hanson

        I’m not saying you shouldn’t try to learn things from our data on self-improving programs and human software development. I’m saying you should try to express those things in terms of standard concepts from economic growth, and compare them to other common growth models, in order to judge how likely they could be to cause foom.

      • AnotherScaryRobot

        We don’t know that anything as elaborate as developing better algorithms is required. The brain is would seem to be significantly resource-limited as a consequence of the caloric energy available in the human ancestral environment. There are also some very real practical physical problems associated with fitting the thing into your head — problems which make humans very susceptible to head injury, and require babies to be born in an extremely premature state.

        The brain contains many repetitive structures. It may well be the case that given an emulated human brain free from the above constraints, simply adding many more copies of these repetitive structures could enhance cognition significantly, with no “intelligent” changes to underlying architecture and no development of new algorithms.

      • http://kruel.co/ Alexander Kruel

        AnotherScaryRobot, note that bigger brains don’t correlate well with intelligence:

        Research suggests that bigger animals may need bigger brains simply because there is more to control — for example they need to move bigger muscles and therefore need more and bigger nerves to move them.

        Also see this article:

        [...] animal brains [...], which can vary in size by more than a hundred-fold—in mass, number of neurons, number of synapses, take your pick—and yet not be any smarter. Brains get their size not primarily because of the intelligence they’re carrying, but because of the size of the body they’re dragging.

        I’ve termed this the “big embarrassment of neuroscience”, and the embarrassment is that we currently have no good explanation for why bigger bodies have bigger brains.

        If we can’t explain what a hundred times larger brain does for its user, then we should moderate our confidence in any attempt we might have for building a brain of our own.

  • Don Geddis

    Robin, it bothers me that whenever you describe the other side, you use terms that seem deliberately implausible: “a cheap unnoticed and initially low ability AI (a mere “small project
    machine in a basement”) could without warning over a short time (e.g., a
    weekend) become so powerful as to be able to take over the world”.

    The problem is, that the issues of a singleton using self-improvement to take over the world, are essentially the same, even if it starts with some major university or military project, and even if it takes years before the “foom”.  Think more like Skynet from the Terminator movies.

    Nothing essentially changes in the concerns, or the outcomes for humanity, if the thing starts with a few more resources, or takes a bit longer.  So it seems to me that you’re deliberately belittling the other side, by always describing it as a loner in a basement, over a weekend.  That just seems an attempt to convince people by appealing to their gut intuitions, but in an inappropriate way.  Because the “loner basement” and “weekend” parts, while powerful for the reader’s skeptical intuition, are not in fact critical parts of the disagreement between you and Eliezer.

    • http://kruel.co/ Alexander Kruel

      I agree with you that Robin is belittling the other side. Nobody would actually hold such absurd beliefs…

      It might be developed in a server cluster somewhere, but as soon as you plug a superhuman machine into the internet it will be everywhere moments later.

      Luke Muehlhauser, CEO of MIRI (formely known as the Singularity Institute)

      Just to be clear on the claim, “fast” means on a timescale of weeks or hours rather than years or decades; and “FOOM” means way the hell smarter than anything else around, capable of delivering in short time periods technological advancements that would take humans decades, probably including full-scale molecular nanotechnology (that it gets by e.g. ordering custom proteins over the Internet with 72-hour turnaround time).

      — Eliezer Yudkowsky, Recursive Self-Improvement

      • VV

        Spot the differences:

        It might be developed in a server cluster somewhere, but as soon as you
        plug a superhuman machine into the internet it will be everywhere
        moments later.

        By the time Skynet became self-aware… it had spread into millions of computer servers across the planet.

        Ordinary computers in office buildings, dorm rooms, everywhere.

        It was software in cyberspace.

        There was no system core. It could not be shut down.

    • VV

      But even if you substitute the “AI made in a basement by a lone boy called Eli, fooming before his parents are back home” scenario to the “AI made by a joint venture between Microgooglebook, AppleBM, the NSA, the CIA and Soviet Russia, fooming over the course of a decade” scenario, the objection stands.

      We have no solid arguments to believe that the ability to modify yourself implies with high probability an intelligence explosion.

  • Abram Demski

    Polite inquiry on behalf of Richard Loosemore: some of his comments seem not to be showing up properly.

  • Michael Vassar

    I actually don’t believe that the empirical and theoretical sides of economic theory relate to one another significantly. Theoretical econ appears too me to perdict foom in general and then to be modified ad-hoc to pretend that it doesn’t.

    • http://overcomingbias.com RobinHanson

      Standard physics equations, when simply projected backwards in time, give very wrong estimates of past features. To get accurate estimates you have to add in an “arbitrary” and hard to adequately formalize constraint that entropy was very low in the distant past. Part of what a physics expert knows is which models to use how in order to get reasonable estimates from them.

      Similarly, part of growth experts know is which models to use how in order to get reasonable growth features. Since it is easy for models to foom but apparently hard for reality to room, they reasonably avoid foom models. Why isn’t that attitude toward models just as reasonable as the physicists’ attitude toward models that estimate the past?

      • VV

         

        Standard physics equations, when simply projected backwards in time,
        give very wrong estimates of past features. To get accurate estimates
        you have to add in an “arbitrary” and hard to adequately formalize
        constraint that entropy was very low in the distant past.

        That sounds strange. Can you please elaborate?

        As far as I know, the second law of thermodynamics is part of the “standard physics equations”. Do you mean reversible physics equations?

        Similarly, part of what growth experts know is which models to use how in order to get reasonable growth features.

        Correct me if I’m wrong, but mathematical models of economic growth typically don’t fit experimental data very well (except perhaps World3, which economists look with disgust). Of course, this doesn’t give Yudkowsky free pass to make things up.

      • http://kruel.co/ Alexander Kruel

        He might be talking about thermodynamic time asymmetry

        See e.g. his post ‘Scandalous Heat‘. Also see the follow-up.

  • Pingback: Alexander Kruel · MIRI/LessWrong Critiques: Index

  • mjgeddes

    Actually I think Yudkowskys right about this one (shame he got everything else wrong though).

    Ultimately, matters will not be resolved by internet debates, but by empirial facts on the ground.   The debate is resolved when there an empirical #win.  So we should emphasize again, the importance of #winning.

    Hackers Maxim #11
    ‘The one true alpha move is #winning’