Debate Is Now Book

Back in 2008 Eliezer Yudkowsky blogged here with me, and over several months we debated his concept of “AI foom.” In 2011 we debated the subject in person. Yudkowsky’s research institute has now put those blog posts and a transcript of that debate together in a free book: The Hanson-Yudkowsky AI-Foom Debate.

Added 6Sept: Bryan Caplan weighs in.

GD Star Rating
loading...
Tagged as: , ,
Trackback URL:
  • http://entitledtoanopinion.wordpress.com TGGP

    I hope this isn’t the “blook” we were supposed to be waiting for from Yudkowsky.

    • Zingram

      Or Robin.

    • Jess Riedel

      No, Yudkowsky has compiled the (broader) material about his views of rationality into a book format, but apparently getting it to a publishable form and style has been more difficult than expected. I suggest hassling Luke Muehlhauser, as I briefly tried to do, to convince him to release it as a free book in its current raw form. Although I’m not sure if there are copyright issues that would then make publishing it as a traditional book in the future more difficult.

      • http://juridicalcoherence.blogspot.com/ Stephen Diamond

        apparently getting it to a publishable form and style has been more difficult than expected

        For one thing, when you put ideas into “publishable form” (read, try to write classic prose [ http://tinyurl.com/cdzotb4 ]) any lack of coherence becomes evident, and outright contradictions are readily spotted–say, for instance, his “infinite set atheism” versus his endorsement of infinite-worlds theories. ( For some elaboration see, for example, “Infinitesimals: Another argument against actual infinite sets” — http://tinyurl.com/kdjm473 .)

      • Jess Riedel

        I think it’s poor taste to use my comment as an opportunity to criticize Yudkowsky’s writing, especially in this thread where it’s completely off topic. But I might as well point out that the worlds of the many world interpretation are emphatically *not* formally infinite.

      • http://juridicalcoherence.blogspot.com/ Stephen Diamond

        But they’re metaphysically infinite–which is the issue.

      • IMASBA

        What is “metaphysical infinity”? The number of worlds in the many worlds interpretation is equal to the size of the phase space of the universe, which is finite if the universe is finite.

        But I get Jess Riedel’s point that this may not be the place for detailed debate about the content of the book.

      • http://juridicalcoherence.blogspot.com/ Stephen Diamond

        The number of worlds in the many worlds interpretation is equal to the size of the phase space of the universe, which is finite if the universe is finite.

        Are there different versions of many worlds providing different answers to this question? I find the following explanation on LW (strongly upvoted):

        How many ink blots are in this picture ?

        For “many worlds”, imagine that evolving in time, with fuzzy borders.

        The Many Worlds Interpretation doesn’t imply a countable number of worlds that suddenly branch, it’s more like a fuzzy continuum; talking of branching worlds and timelines is just a high-level abstraction that makes things easier to discuss.

        http://lesswrong.com/lw/8ue/how_many_worlds/

        This is to say, the infinity isn’t countable but uncountable, like the number of geometric points said to lie on a line segment.

        I’ve seen this enough on LW to feel confident it’s the dominant interpretation there, but I have also seen, elsewhere, claims like yours, concluding there might be finite worlds.

        By “metaphysical” I mean the theory’s ontological commitments (as opposed to whether it formally invokes infinities). [I'm not a Quinean about the ontological commitments of theories, btw.]

        I’m interested in what’s involved in this disagreement, but also, note that my point here was about “infinite worlds” theories, not just MWI. For example, the Tegmark view that Yudkowsky endorses. Or the use of infinite worlds to create anthropic explanations.

        But I get Jess Riedel’s point that this may not be the place for detailed debate about the content of the book.

        I understand the need not to have a deluge of posts on this tangent, but that problem has been ingeniously solved by Robin with his max. 3 visible posting rule. Am I missing something in Jess Riedel’s complaint?

      • IMASBA

        “Are there different versions of many worlds providing different answers to this question?”

        “This is to say, the infinity isn’t countable but uncountable, like the number of geometric points said to lie on a line segment.”

        It depends on whether you believe states can change over infinitesimal timespans. It’s very much possible time itself is quantized and even if it isn’t it seems to be that there’s, at least on average, a finite time required between state changes (a clue is that after the collapse of a wavefunction following a measurement it needs a measurable amount of time to become a spread out wavefunction again). So it seems to be that there are only a finite number of worlds in the many world interpretation IF the size of the universe is finite (which we just don’t know).

      • VV

        It also depends on whether spatial dimensions are quantized, which is currently unknown.

      • IMASBA

        Aren’t spatial dimensions essentially quantized for the purpose of counting alternative worlds because position accuracy is limited by energy and it seems particles above a certain high energy are so rare (or may even not exist at all) we have never seen one and aren’t likely to even in the vicinity of a gamma ray burst (though of course this isn’t a fixed grid like you would have when space itself was naturally quantized)?

      • VV

        Due to deterministic chaos, arbitrarily small differences in position and momentum are amplified in relatively short amounts of time.

      • IMASBA

        But differences in position and momentum can’t be arbitrarily small, or if they can it’s incredibly rare.

      • VV

        Why?
        If spatial dimension are continuous, then you would generally expect the wavefunction to be spread over a dense infinite set of both basis vector both in the position basis and in the momentum basis.

        (If I understand correctly, under some technical conditions you could write the wavefunction w.r.t. a countable basis, but it would be still infinite unless the quantum system has finitely many degrees of freedom)

      • IMASBA

        “Why?
        If spatial dimension are continuous, then you would generally expect the wavefunction to be spread over a dense infinite set of both basis vector both in the position basis and in the momentum basis.”

        The higher values do not realistically get realized and perhaps never at all. For fermions this means there can only be a finite anumber of them in a finite volume. Bosons can overlap but strange things will happen if too many of them overlap (they all interact with gravity). Of course infinity times something really small is still infinity, but that only goes to show that you will not have infinitely many worlds without some inherent infinities within our universe (infinite size of our universe, points with infinite density, particles with infinite energy), so one does not have to believe in infinite quantities to believe in many worlds

      • VV

        You don’t need infinitely many particles in order to have an infinitely dimensional Hilbert state-space

      • IMASBA

        Indeed you don’t, infinite energy ranges or infinite densities also do the trick, it’s just one way to get to an infinite number of possible worlds, but you’re definitely not going to get infinitely many worlds without allowing for a type of infinity within every world.

      • http://juridicalcoherence.blogspot.com/ Stephen Diamond

        you’re definitely not going to get infinitely many worlds without allowing for a type of infinity within every world.

        Doesn’t that contradict your earlier claim:

        “It depends on whether you believe states can change over infinitesimal timespans.”

        I think what you’re saying is that physical continuity (nonquantization of time or space) implies that infinitisimal time spans exist. (I think assuming continuity implies actual infinitesimal time spans create Zeno’s paradox – http://tinyurl.com/b9kn4tb )

      • IMASBA

        “Doesn’t that contradict your earlier claim:”

        Well, a process that changes infinitely fast is a bit like an infinite density in time…

        “I think what you’re saying is that physical continuity (nonquantization of time or space) implies that infinitisimal time spans exist.”

        Yes, but even if infinitesimal time spans exist that does not have to mean any physical process works in infinitesimal time steps.

        “I think assuming continuity implies actual infinitesimal time spans create Zeno’s paradox”

        Or processes just take an infinite number of infinitesimal steps at a time.

        This reminds me of an Isaac Asimov short story where people have discovered how to travel to alternative universes. They reckon there are an infinite number of them so people start claiming alternative Earths for themselves and they expect to never encounter an alternative humanity. Than one day they do because the alternative Earths with humans on them are a fraction of all the worlds: the probability of meeting them is actually finite. Perhaps nature solves Zeno’s paradox in a similar way.

  • Zingram

    Someone know when Robin’s book is coming out?

  • jack master

    Off topic… When will your book be available?

  • Will Sawin

    Summary: Your model of growth rates seems to predict that in a period of about 6 years after the em revolution, life on Earth would change to some nearly unrecognizable form and economic growth of several orders of magnitude would occur. It seems unlikely in such a scenario that current humans would survive in a way that current humans deem acceptable. If/when ems become imminent, so this affects nearly all living humans, would you then say that em researchers were engaging in total war against the rest of humanity?

    Elaboration: In your model, development of life/the economy/civilization proceeds in exponential growth periods punctuated by jumps in the growth rate. The length of these growth periods is itself decreasing exponentially, by a factor of 50 – 300 or so. So if the industry period lasts 250 years or so, the next period should last not much more than 5 years, and the next not much more than 1 month, and the next not much more than 1 day. Presumably this process stops at some point! But still we should expect a few more growth speedups.

    After a few such speedups, the end result is unlike anything that came before it and predictions become incredibly difficult, if not impossible. In particular, it seems tenuous to predict one’s own survival. How confident should a pre-brains animal species be that they will survive farming, or industry – what would that even mean? How confident should an animal with brains be that it will survive industry, or world takeover by ems? Many such animals would find this confidence misplaced. Will ems with an economic doubling time of one month leave current hunter-gatherers with vague property rights to the land they use untouched or integrate them as trade partners? It seems far more likely that they will find some excuse to expropriate the land to far more productive purposes. The hunter-gatherers would die or be forced into an existence they do not find very agreeable.

    • http://overcomingbias.com RobinHanson

      Your comments have little to do with my particular analysis, and apply to pretty much any scenario of rapid change. The faster that change happens, the less confidence those adapted to prior regimes can have that they or their descendants will continue to be adaptive at any given time distance in the future. If you think of researchers as causing change, you might claim that researchers in general are warring against everyone who wants confidence in continuing. I don’t have any special expertise in such metaphorical uses of the “war” concept, so I don’t see why I should bother to have a position on that topic.

      • Will Sawin

        In the debate, you described the possibility of an AI built by Eliezer reshaping the world in a period of 2 weeks as total war. The key feature of total war here being that when organizations believe it might happen they will fight viciously and brutally with other organizations to ensure it doesn’t happen or happens in a way that’s agreeable to then.

        Do you think ems built by a corporation reshaping the world in a period of 6 years does not have that key feature?

      • http://overcomingbias.com RobinHanson

        If I send tanks into your nation to physically take control, that is war. If I invent a new kind of car likely to take away customers from your car factory, that isn’t war. But that is just how I see the word “war.” As I said, I’m no expert in word usage.

      • Will Sawin

        Or:

        I create a new set of technologies that allow me (or others I create) to make vastly more productive use of your land then you do. Claiming “manifest destiny”, We kill you and take your resources, and create a legal system that allows us to do so safely.

        Or:

        I create a new set of technologies which leads to a further technological revolution until you are to us as pandas are to you. We take over your habitat and turn it into housing for trillions of ems. We make some attempt to preserve you in a zoo, but it is insufficient, and we are not willing to give up the resources to do better.

        Or:

        After several technological revolutions, you are to us as insects are to you. We destroy you with extreme prejudice.

        Or:

        After many technological revolutions, you are so slow-moving as to be seen as a mountain rather than a person. You are mined for natural resources.

        ———-

        In creating ems and bringing forth a massive speedups in the rate of economic growth, all of these seem like real possibilities. “I don’t know whether the tank I am building will kill you or not” is not a comforting defense.

      • IMASBA

        Will Sawin already provided some excellent rebuttals, but we can look at it from less extreme angles. Is it war when your technology makes me starve to death (because you also crusaded against welfare and retraining vouchers) and I riot violently in return against the government you bought? It may not be an official war but it sure would be a bloody, possibly even genocidal affair.

    • Alexander Gabriel

      I think a more fathomable scenario than a singularity that tests these same considerations is human genetic enhancement. E.g.,

      http://lesswrong.com/lw/ia3/new_article_on_in_vitro_iterated_embryo_selection/

      It seems plausible that we will in a few decades be able to asexually create humans much more capable than those born naturally. Then there would be a question of whether people care enough about the risks of creating superior, conscious creatures to ban it globally. If not there might be a competition where nations raise a generation of borderline posthumans to help them win, either militarily or in the labor market.