Evolved Desires

To a first approximation, the future will either be a singleton, a single integrated power choosing the future of everything, or it will be competitive, with conflicting powers each choosing how to perpetuate themselves.  Selection effects apply robustly to competition scenarios; some perpetuation strategies will tend to dominate the future.  To help us choose between a singleton and competition, and between competitive variations, we can analyze selection effects to understand competitive scenarios.  In particular, selection effects can tell us the key feature without which it is very hard to forecast: what creatures want.

This seems to me a promising place for mathy folks to contribute to our understanding of the future.  Current formal modeling techniques are actually up to this task, and theorists have already learned lots about evolved preferences:

Discount Rates: Sexually reproducing creatures discount reproduction-useful-resources given to their half-relations (e.g., kids, siblings) at a rate of one half relative to themselves. Since in a generation they get too old to reproduce, and then only half-relations are available to help, they discount time at a rate of one half per generation.  Asexual creatures do not discount this way, though both types discount in addition for overall population growth rates.  This suggests a substantial advantage for asexual creatures when discounting is important.

Local Risk: Creatures should care about their lineage success, i.e., the total number of their gene’s descendants, weighted perhaps by their quality and relatedness, but shouldn’t otherwise care which creatures sharing their genes now produce those descendants.  So they are quite tolerant of risks that are uncorrelated, or negatively correlated, within their lineage.  But they can care a lot more about risks that are correlated across such siblings.  So they can be terrified of global catastrophe, mildly concerned about car accidents, and completely indifferent to within-lineage tournaments.

Global Risk: The total number of descendants within a lineage, and the resources it controls to promote future reproduction, vary across time.  How risk averse should creatures be about short term fluctuations in these such totals?  If long term future success is directly linear in current success, so that having twice as much now gives twice as much in the distant future, all else equal, you might think creatures would be completely risk-neutral about their success now.  Not so.  Turns out selection effects robustly prefer creatures who have logarithmic preferences over success now.  On global risks, they are quite risk-averse.

Carl Shulman disagrees, claiming risk-neutrality:

For such entities utility will be close to linear with the fraction of the accessible resources in our region that are dedicated to their lineages. A lineage … destroying all other life in the Solar System before colonization probes could escape … would gain nearly the maximum physically realistic utility … A 1% chance of such victory would be 1% as desirable, but equal in desirability to an even, transaction-cost free division of the accessible resources with 99 other lineages.

When I pointed Carl to the literature, he replied:

The main proof about maximizing log growth factor in individual periods … involves noting that, if a lineage takes gambles involving a particular finite risk of extinction in exchange for an increased growth factor in that generation, the probability of extinction will go to 1 over infinitely many trials. … But I have been discussing a finite case, and with a finite maximum of possible reproductive success attainable within our Hubble Bubble, expected value will generally not climb to astronomical heights as the probability of extinction approaches 1.  So I stand by the claim that a utility function with utility linear in reproductive success over a world history will tend to win out from evolutionary competition.

Imagine creatures that cared only about their lineage’s fraction of the Hubble volume in a trillion years.  If total success over this time is the product of success factors for many short time intervals, then induced preferences over each factor quickly approach log as the number of factors gets large. This happens for a wide range of risk attitudes toward final success, as long as the factors are not perfectly correlated. [Technically, if U(prodtN rt) = sumtN u(rt), most U(x) give u(x) near log(x) for N large.]

A battle for the solar system is only one of many events where a lineage could go extinct in the next trillion years; why should evolved creatures treat it differently?  Even if you somehow knew that it was in fact that last extinction possibility forevermore, how could evolutionary selection have favored a different attitude toward such that event?  There cannot have been a history of previous last-extinction-events to select against creatures with preferences poorly adapted to such events.  Selection prefers log preferences over a wide range of timescales up to some point where selection gets quiet.  For an intelligence (artificial or otherwise) inferring very long term preferences by abstracting from its shorter time preferences, the obvious option is log preferences over all possible timescales.

Added: To explain my formula U(prodtN rt) = sumtN u(rt),

  • U(x) is your final preferences over resources/copies of x at the “end,”
  • rt is the ratio by which your resources/copies increase in each timestep,
  • u(rt) is your preferences over the next timestep,

The righthand side is expressed in a linear form so that if probabilities and choices are independent across timesteps, then to maximize U, you’d just pick rt to max the expected value of u(rt ).  For a wide range of U(x), u(x) goes to log(x) for N large.

GD Star Rating
loading...
Tagged as:
Trackback URL:
  • http://www.transhumangoodness.blogspot.com Roko

    Robin: ” Evolved Desires …

    …or it will be competitive, with conflicting powers each choosing how to perpetuate themselves. Selection effects apply robustly to competition scenarios; some perpetuation strategies will tend to dominate the future.”

    Now, it seems obvious to me that most people do not want a world which is full of creatures who want to propagate themselves at all costs. We do not want to live in the crack of a future dystopia. When I say it seems obvious, I mean that no sophisticated mathematical analysis of maximally competitive scenarios is required to ascertain their ultimate undesirability.

    Why do I claim that there is no need for such analysis? Because the underlying assumption for these “evolution of values under intense competition” scenarios is that you have to only care about perpetuating yourself, at absolutely all costs. Empirical fact: this is not a desirable state to be in, as far as the vast majority of people on this planet are concerned.

    Let me define a limited competition scenario to be one where selection effects do not apply robustly. It seems to me that the only vaguely acceptable options are either a singleton or a fairly limited competition like the competition that people in developed nations with a welfare state find themselves in.

    I think that the only really interesting question here is how stable a limited competition system can be, and how increasing technology affects this stability. Most people in the transhumanist sphere seem to be arguing that we need a benevolent singleton because limited competition scenarios are increasingly unstable as technology increases – they will lead to either an extinction event or a total competition scenario. I’d be interested to see this assumption challenged.

  • http://hanson.gmu.edu Robin Hanson

    Roko, how can I challenge an assumption until arguments are offered in its favor? There are gains from trade and from peace even for evolved creatures; we are such creatures and we understand this about our world; why couldn’t they? Treating a competition as a total war is only required when you expect others to treat it that way; give peace a chance.

  • http://reflectivedisequilibria.blogspot.com/ Carl Shulman

    “If total success over this time is the product of success factors for many short time intervals… A battle for the solar system is only one of many events where a lineage could go extinct in the next trillion years; why should evolved creatures treat it differently?”

    What sort of factors are you thinking about for a singleton expanding into our limited and apparently uninhabited accessible region, with current physical limits (thermodynamics, no FTL, etc) assumed? Are you thinking about the entities’ credence in the hypothesis that resources can increase vastly beyond those that physical limits seem to suggest? If resources could grow indefinitely, e.g. if there was a technological way to circumvent the laws of thermodynamics, then entities with unbounded utility functions (whether linear or log in reproductive success) will all have their calculations dominated by that possibility, and avoid struggles in the solar system that reduce their chances of getting access to such unbounded growth. I’m planning to talk more about that, but I started off with an assumption of common knowledge of current physics to illustrate dynamics.

    “There cannot have been a history of previous last-extinction-events to select against creatures with preferences poorly adapted to such events.”

    Intelligent, foresightful entities with direct preferences for total reproductive success will mimic whatever local preferences would do best in a particular situation, so they won’t be selected against, but in any case where the environment changes so that evolved local preferences are no longer optimal, those with direct preferences for total success will be able to adapt immediately, without mutation and selection.

  • http://www.transhumangoodness.blogspot.com Roko

    Without wishing to put words in his mouth, Michael Anissimov seems to be arguing in favor of a singleton for exactly these reasons in his latest post:

    http://www.acceleratingfuture.com/michael/blog/2008/11/the-challenge-of-self-replication/

    He doesn’t seem to consider strong competitive scenarios as a serious possibility, so he sees the future as a choice between a singleton and extinction. [Correct me if I’m wrong here, Michael]

    Now, what arguments can I think of in favor of the instability of a limited competition scenario as technology increases? Well, one very general argument that has been aired many times is Eli’s idea that we human beings are adaption executers, we execute a set of adaptions that used to maximize our inclusive genetic fitness in our EEA. Since the environment has changed a lot, we are actually *not* fitness maximizing creatures any more. This defeats the following argument you made:

    There are gains from trade and from peace even for evolved creatures; we are such creatures and we understand this about our world; why couldn’t they?

    In order to get a qualitative idea of what a future full of genuinely fitness maximizing creatures will be like, we should look at natural environments where extant creatures genuinely maximize their genetic fitness. Nature red in tooth and claw.

    **********************

    When we talk about “giving peace a chance” in a *competitive* scenario that is *not a singleton*, we must be careful not to assume both a statement and it’s negation. From Nick Bostrom’s paper:

    “All the possible singletons share one essential property: the ability to solve global coordination problems”

    So, peace is possible in a non-singleton scenario if and only if it is in each agent’s interest to peaceful. However, if we combine this with the basic axiom of a competitive scenario:

    “Different agents have conflicting goals”

    We get a global co-ordination problem that, by assumption, cannot be solved, and therefore it is not in each agent’s interest to be peaceful.

    The idea of a peaceful, competitive non-singleton scenario is self-contradictory.

  • James Andrix

    Selection effects apply robustly to competition scenarios; some perpetuation strategies will tend to dominate the future.

    That seems effectively equivalent to a singleton, and an unfriendly one at that. (It does not care about us.) This is the big thing you think will shape the future, that almost no one will be able to get away from.

    There are gains from trade and from peace even for evolved creatures; we are such creatures and we understand this about our world; why couldn’t they?

    Will we implement this understanding before or after hand held superweapons? I would really prefer a _better_ world for the next billion years, not just one with FASTER MORE POWERFUL monkeys fighting over stuff.

  • http://profile.typekey.com/EWBrownV/ Billy Brown

    “Current formal modeling techniques are actually up to this task…”

    This is an interesting research proposal, but the models used to investigate natural evolution make some assumptions that don’t hold for scenarios about advanced civilizations. In particular:

    1) Older generations do not automatically die off in some characteristic time. Instead they suffer a very low annual mortality that can be reduced to arbitrarily low levels by investing resources in survivability rather than reproduction.

    2) Individuals can reproduce at any age, at a rate limited only by their available resources.

    3) Rather than using resources to reproduce, many kinds of individuals can simply grow in size instead (i.e. an AI with thousands of bodies controlled by a single distributed consciousness).

    4) Rather than having fixed preferences engineered by evolution, individuals can explicitly model the results of various strategies and switch from one to another as changing circumstances dictate.

    It seems obvious that, while there would still be selection pressures in such a civilization, the traditional concepts of “species”, “individual” and “generation” don’t exist in anything like the same form. Building a model that can handle this situation does seem feasible, but it would be a big departure from the ones that currently exist.

  • http://shagbark.livejournal.com Phil Goetz

    Local Risk: Creatures should care about their lineage success, i.e., the total number of their gene’s descendants, weighted perhaps by their quality and relatedness, but shouldn’t otherwise care which creatures sharing their genes now produce those descendants. So they are quite tolerant of risks that are correlated, especially negatively, within their lineage. But they can care a lot more about risks that are correlated across such siblings. So they can be terrified of global catastrophe, mildly concerned about car accidents, and completely indifferent to within-lineage tournaments.

    Huh?

    What does “risks that are correlated, especially negatively, within their lineage” mean? And what is the connection between that, and global catastrophe?

    Also, what does “logarithmic preferences over success now” mean? That preference is proportional to the logarithm of… what? Not “success now”, surely, since we are contrasting short-term success with longer-term success. Again, what is the connection with global risks?

  • http://shagbark.livejournal.com Phil Goetz

    Technically, if U(prodtN rt) = sumtN u(rt)

    Please explain what U, t, N, and u are.

    How did you get subscripts and superscripts? HTML doesn’t work.

  • http://hanson.gmu.edu Robin Hanson

    Carl, you lost me. Your first quote of me isn’t talking about a singleton, and I don’t see how physics knowledge is relevant. On your response to your second quote of me, you can’t just assume you know what sort of risk-aversion regarding the final outcome is the “true” preferences for “total success.” If evolution selects for log preferences on all timescales on which it acts, why isn’t log risk aversion the “true” total success risk-aversion?

    Roko, just repeating the phrase “Nature red in tooth and claw” is just nowhere near a sufficient argument.

    James, will you really argue detailed analysis is irrelevant because you just know that more powerful monkeys are always bad things?

    Billy, yes some details change; that is why I’m calling for more analysis, so we can see how important those details end up being.

    Phil, If there are two possible outcomes, 1 for me and 0 for you, or 0 for you and 1 for me, our payoffs are negatively correlated. The formula should define everything. My browser (Firefox) shows its sub-sups in the post, but not in your comment; no idea why.

  • http://reflectivedisequilibria.blogspot.com/ Carl Shulman

    I’ll reply in a post.

  • http://shagbark.livejournal.com Phil Goetz

    I still don’t see the connective tissue in your argument. A car accident and a global catastrophe are both correlated positively across siblings; how do your comments imply that they should be viewed differently?

    “The formula should define everything.”

    The formula doesn’t say what U, u, or r are. There is really and truly nothing in your post to indicate what U and u are. Some sorts of utilities, perhaps; but how do they differ? r is probably some measurement of risk, but I need more information, as it does not make sense to make utility a pure function of risk.

  • http://profile.typekey.com/sentience/ Eliezer Yudkowsky

    Robin: If evolution selects for log preferences on all timescales on which it acts, why isn’t log risk aversion the “true” total success risk-aversion?

    Entities with logarithmic preferences over their aggregate number of copies in total world-histories should behave sublogarithmically when making local, independent choices on the next generation. The evolutionary analysis similarly talks about entities that you are likely to see in the sense of their being most frequent, not entities whose logarithms you are likely to see.

    You can’t literally have logarithmic preferences at both global and local timescales, I think. If global preference is logarithmic, wouldn’t local preference be log-log?

    Anyway: would you agree that: a linear aggregate utility over complete world-histories corresponds to logarithmic choices over spatially global, temporally local options, whose outcome you believe to be uncorrelated to the outcome of similar choices in future times.

  • James Andrix

    Robin:
    They don’t have to be always bad things, they are not reliably good things. Have we achieved friendliness yet?

    Detailed analysis of potential competitive futures is great. I’m not saying this isn’t important work. I AM saying detailed analysis isn’t important to the question of if we need to do major work on friendliness. It may be VERY important to the question of how to obtain friendliness if Eliezer is wrong about the FOOM.

    We know evolution isn’t friendly.

    Your argument seems to be that the bulk of the universe’s resources will go to entities who are best at snatching them up (either for themselves or for their descendants or kin), and we can tell certain things about those entities attitudes towards resources and kin and risk.

    You seem to be arguing that the universe will be dominated my beings that want to take as much of the universe as they can (or their descendants wanted it for them.) and care mostly about those most closely related to them.

    These are not compatible with my values. ESPECIALLY if I’m not closely related to those entities, and much less powerful.

    My point is that YOU are arguing that the future will be filled with super entities entities that would be unfriendly to me, and you, and I think the bulk of humanity (both now as it is now, and as it would be then). Our only advantage against them is that we are in their past.

    So we need to either to work on a friendly singleton if a singleton is feasible, work on imposing further selection effects to encourage friendliness (using evolution as a singleton) or somehow ensure that all the initial competitors will be friendly, so that there are no unfriendly lineages.

    What past trends point to a peaceful future at the hands of evolution and super-monkeys?

    (Actually, The entities described seem more like von Neumann probes with extra personality. What will entities evolved to be colonizers do when the universe is full? Why should I value that future?)

  • http://hanson.gmu.edu Robin Hanson

    Eliezer, I think you are just mistaken; log preferences aggregate or split in time to log preferences. Regarding your last question, I said a wide range of preferences over final outcomes, including linear preferences, converge to log preferences over each step.

    Phil U(x) is your final preferences over resources/copies of x at the “end.” r_t is the ratio by which your resources/copies increase in each timestep. u(r_t) is your preferences over the next timestep, expressed in a linear form so that if probabilities and choices are independent across timesteps, then in order to maximize U, you’d just make your choices about r_t maximize the expected value of u(r_t).

  • http://profile.typekey.com/sentience/ Eliezer Yudkowsky

    Eliezer, I think you are just mistaken; log preferences aggregate or split in time to log preferences.

    Ah, okay, I see my problem. I was assuming that taking the log of population sizes just put us into a log-world, exchanging multiplication for addition. But in the new world, options add fixed amounts to your current total, regardless of your initial position, so preferences are just aggregative (not logarithmic) in the new world.

    (Thinks.)

    I think what this reveals is that for repeatable choices with a certain kind of temporal independence and an indefinite time horizon, your local preferences will start corresponding to a representation under which the effect of those choices is purely aggregative, if such a representation exists. A representation where -4 units of negative is exactly balanced by +1 and +3 positive outcomes. As your time horizon approaches the indefinite, such an approach will dominate.

    If you expect to encounter lots of options with non-multiplicative effects – like “this will square my population, this will take the square root of my population” – then you’ll be wise to regard those as +1 and -1 respectively, even though a logarithmic analysis will call this +X vs. -0.5X.

  • Tim Tyler

    My point is that YOU are arguing that the future will be filled with super entities entities that would be unfriendly to me, and you, and I think the bulk of humanity (both now as it is now, and as it would be then). Our only advantage against them is that we are in their past.

    So we need to [snip plan]

    It sounds as though you want to block the natural evolutionary process – turning yourself into a parasite on the glorious beings of the future, for your own benefit. Enslaving our offspring may be possible – but do we really want to screw over our own legacy in that way? What will “human friendliness” do for civilization’s spaceworthyness? How long do you think you would be prepared to keep development bogged down in the human phase for?

  • http://hanson.gmu.edu Robin Hanson

    Eliezer, it sounds like you are probably right with your ending comment, though it could be interesting to hear it elaborated, for a wider audience.

  • http://profile.typekey.com/halfinney/ Hal Finney

    It’s interesting that this aspect of the debate comes down to whether people’s utility functions are (approximately) logarithmic, and equivalently whether they are risk averse or risk neutral. The paper Robin cites seems to show that evolution should gift us with logarithmic utilities, making us risk averse. And indeed the psychological literature largely confirms this. The basic argument in the paper is that the overriding consideration built into us from evolution is avoiding extinction. As the author notes: “survival must have been a dominant ability, because among the trillions of predecessors of a person living today, from the first cell to his own parents, there wasn’t a single one that died before reproduction age.” We are built to avoid destruction rather than to maximize success.

    Billy Brown asks whether we can’t just override this, for example deciding that risk neutrality is better. On one hand it sounds almost incoherent to propose changing our preferences, since by definition our preferences are what we try to satisfy. At the same time, it does seem that we perceive a contradiction between our desires for success and the timidity which risk aversion imposes on us. Maybe this is an example of evolution screwing up, as it did with sex, and using too many layers of indirection in imposing the desired behavior. Rather than making us want to reproduce, it makes sex feel good; but we can separate the two. In the same way, rather than making us value log(success) directly, it makes us want success but afraid when we take risks; yet we can find ways to overcome the fear.

    In the end though I wonder if it is true that the shape of this utility function will decide and determine whether the future will include an attempt to grab for all resources available, wiping out all competitors. Are risk neutral entities fated to wage all out war even when the probability of success is low, due to the great rewards which might be gained? Are risk averse entities automatically going to be more interested in trade and mutual cooperation by virtue of their logarithmic utility functions? It seems odd for this fundamental fact about the future to turn on such a specific and seemingly esoteric technical matter.

  • Z. M. Davis

    Roko: “We get a global co-ordination problem that, by assumption, cannot be solved […] The idea of a peaceful, competitive non-singleton scenario is self-contradictory.”

    I’m pretty sure this is begging the question. Why assume the coördination problem can’t be solved? If none of the competing entities can overpower the others, then they may well find it beneficial to trade or coöperate peacefully. Sure, each might prefer to take over entirely, but when that option isn’t on the table, an eternal stalemate (peace) could be the next best thing. Of course, said peace could still end up destroying everything we value.

    I don’t see how the “adaptation-executers, not fitness maximizers” point is relevant (and for the record, the citation should really go to Cosmides and Tooby, not Eliezer). I think Robin’s point is that humans do compete (e.g., in markets) without slaughtering each other; that we’re competing for other things than fitness doesn’t really change the argument.

    Tim Tyler: “[B]ut do we really want to screw over our own legacy in that way?”

    Yes. You seem to be suggesting that a future filled with “Nietzschean” fitness-maximizers racing to burn the cosmic commons would be “glorious”–but why? That scenario has absolutely nothing to do with sorting pebbles into correct heaps–which, as we all know, is the only thing that matters.

  • James Andrix

    Tim:
    This is as much about them as it is us. Robin portrays a world in which ems may be routinely deleted because they are no longer economical. Enslavement and murder of what are essentially humans is an outcome I want to prevent. Giving the universe to whatever the most brutally efficient uploads are is just the undesirable situation most directly suggested by this post.

    Not murdering EMs to make more EM’s is not enslaving our offspring. I don’t think making AI friendly as opposed to unfriendly is either. In any case, DEVELOPMENT doesn’t have to be bogged down much at all. Our current resource needs are small, compared to what is available

    I’m not precluding that a Friendly AI will forcibly upload and enhance everyone, either. I’d prefer people have options but a FAI might know better.

  • http://causalityrelay.wordpress.com/ Vladimir Nesov

    Even a war between morally opposed sides could be resolved by cooperating in one-off prisoner’s dilemma, with both sides choosing to avoid the war that isn’t expected to be reliably won by either side, incorporating the eternal truce in their morality, so that they won’t need to worry about either hot or cold war in the future, and will just go about their business like each of them is alone in the universe.

  • Nick Tarleton

    Tim: (I don’t know if I should belabor this point, but I don’t think anyone’s brought up yet the fact that) a human-Friendly AI could rapidly colonize the universe with pure AI probes (with stable goal systems and defenses against mutation, to prevent commons-burning), even if all humans stayed biological (which is, of course, a ridiculous distortion of Friendliness).

  • Josh Witten

    “Selection effects apply robustly to competition scenarios”

    This depends on the number of entities competing and the strength of selection. At selection coefficients approximately equivalent to the inverse of the effective population size, chance events will dominate the dynamics.

  • Douglas Knight

    So they are quite tolerant of risks that are correlated, especially negatively, within their lineage.

    I find that statement quite confusing. I can see two ways out: one is that “lineage” means descendants and excludes siblings.

    The other is that it is some kind of typo. I would change it to: “they are quite tolerant of risks that are not correlated, or negatively correlated; but they are sensitive to risks that are positively correlated across siblings.”

  • James Andrix

    Also, Tim:
    Any arguments about what we want to do to our offspring is appealing to the exact same morality that the Friendly AI will have. Yes there could be situation where the AI would kill many people to save many more, if our morality accepts that.
    The AI will also weigh to good of the spaceworthyness of civilization against other conflicting interests. Those conflicting interests will be other kinds of things WE would care about. Whatever its decisions, it will have good reasons that both of us would likely agree with, if we understood the particular situation.

    Building an AI that is not friendly is essentially acting as if you don’t care about something you care about.

    There is a bit of rhetorical judo here. As it is defined, you can’t make any compelling argument that a FAI would behave in any way we would view as a downside. As soon as I view a course of action as not the best, it also becomes what I would predict the FAI NOT to do.

    I’m not sure if this represents a problem with how FAI is defined.

  • michael vassar

    Robin: I usually criticize economics for its assumption of non-myopic agents and here I see you casually discarding that assumption and assuming myopic agents when it supports you in an argument in a situation which involves non-humans who should be able to self-modify to actually be non-myopic. Does this really constitute a good faith effort to give the argument that seems best justified by the evidence rather than to argue for a position? Everyone agrees that the adversarial process doesn’t result in agreement among disputants, so taking a side in this manner seems contrary to the supposed purpose of this experiment.

    Hal: It seems to me that the future of the universe hinging tightly on what casually seem like technical questions is exactly what we should expect a-priori given that our casual intuitions aren’t very reliable outside of their domain or origin.

  • http://hanson.gmu.edu Robin Hanson

    Hal, if we can understand whatever the future does depend on, why shouldn’t math models help that understanding?

    Josh, yes, agreed.

    Douglas, yes that was a typo; fixed.

    Michael, I’m not assuming myopia.

  • Tim Tyler

    Re: glorious competitive future: I don’t know whether self-directed evolution is either likely or desirable.

    Re: immoral future: I tend to think that the shifting moral zeitgeist means that our descendants will be more moral than us. There will probably always be dead wood that needs discarding. Death seems bad today because it takes with it a library of memories – but in the future, memories will be able to be rescued – if they are deemed worthwhile.

    Re: probes that can’t evolve: their genes would have to be protected – and it would be necessary to ensure that they could not spawn new cultural replicators – lest a new kind of evolution take hold and usurp that of its creators. Crippleware probes would be better than allowing evoulution on other planets to continue without monitoring – but worse then sending proper creatures.

    Having superintelligence wipe us out is only one way to go wrong. Building a whole bunch of things about humans into the utility function of an AI seems like an obvious mistake to me – but perhaps it needs pointing out that attempts to build in dependencies on legacy systems can subsequently go on to cause problems.

  • http://www.transhumangoodness.blogspot.com Roko

    Z.M. davis: “I’m pretty sure this is begging the question. Why assume the coördination problem can’t be solved? If none of the competing entities can overpower the others, then they may well find it beneficial to trade or coöperate peacefully. Sure, each might prefer to take over entirely, but when that option isn’t on the table, an eternal stalemate (peace) could be the next best thing. Of course, said peace could still end up destroying everything we value.”

    Well, indeed. Consider a world populated by a set of agents who all spend all of their available time and energy on elaborate defenses to prevent other agents from killing them, and suppose further that these defenses are almost universally effective. [So we are in a world where defense is much stronger than offense]. It must be the case that if a particular agent deviates from the strategy of spending almost all of his time and energy on defense, he will quickly be killed by other agents: this is the assumption of evolved desires through competition.

    This could all be the case without any global coordinator like a world state, so we are not in the singleton scenario. But one has to ask: does this scenario really count as peace? Does it count as a solved coordination problem? (my answer: no) I certainly agree with you that it doesn’t count as good(!), which means that we agree on the relevant point: Robin is busy analyzing the grim details of scenarios that we don’t want to happen.

    Vladimir Nesov: “with both sides choosing to avoid the war that isn’t expected to be reliably won by either side, incorporating the eternal truce in their morality, so that they won’t need to worry about either hot or cold war in the future, and will just go about their business like each of them is alone in the universe.”

    If those two sides constitute all of the agents in the world, such an “eternal truce” obtained by rewriting each others’ moralities is another way of saying “singleton”.

    James Andrix: Your argument seems to be that the bulk of the universe’s resources will go to entities who are best at snatching them up (either for themselves or for their descendants or kin), and we can tell certain things about those entities attitudes towards resources and kin and risk. You seem to be arguing that the universe will be dominated my beings that want to take as much of the universe as they can (or their descendants wanted it for them.) and care mostly about those most closely related to them. These are not compatible with my values…

    – Exactly. I agree completely. If this is obvious to me, James and (I think ) to Z.M., why is it not obvious to everyone? Perhaps I am making some huge error here, which I’d be obliged if someone could point out. Perhaps I should spell out my argument again:

    1. Evolved desires assumption < ==> the vast majority of future agents will care only about propagating themselves, to the exclusion of all else.

    2. Any world satisfying this assumption is not a world I want to live in, because I do not consider such beings to be value-bearing entities.

  • http://hanson.gmu.edu Robin Hanson

    I gotta say it is kinda creepy how many commentators here at OB are so eager to build a God to rule us all that they cannot imagine wanting any other world, instead of because some tech problem forces us to choose between making a God and extinction. Does feel like cult-land.

  • http://causalityrelay.wordpress.com/ Vladimir Nesov

    @Roko:
    Probably, efficient world order is always in a singleton, with all “local” activity optimized with respect to the rest of the world. For example, Friendly singleton needs to initially portion the moral pie among slightly different individual humans, the situation is similar. “Singleton” is not so much one of the many possible ways of building the future, but a way to characterize optimal solution in general.

  • http://www.transhumangoodness.blogspot.com Roko

    Robin: “they cannot imagine wanting any other world,”

    I can imagine wanting other kinds of worlds. I’d be quite happy with some of the following non-singleton worlds:

    1. Worlds in which agents have conflicting desires, but are non-rational in a certain kind of innocent way: they simply don’t realize that they can satisfy their desires by screwing other people over. There is some legend about an isolated community of pacific islanders who did not realize that it was possible to lie until western civilization arrived. Global co-ordination problems would never arise here.

    2. Worlds in which agents can have conflicting desires, but by some lucky coincidence they never do. Global co-ordination problems would never arise here.

    3. Worlds in which global co-ordination problems never escalate beyond the trivial level for some odd reason. For example in this world agents argue about whether to have meat or fish for dinner, but they don’t have wars or kill each other. These kinds of worlds would (ethically) be like living in a children’s book, everything would always turn out alright in the end. Clearly there is a whole continuum here – there exist worlds in which there is some medium level of conflict, perhaps involving shouting, fights, lying, defamation, mild hatred, etc but no killing.

    Note also that the term “singleton” is somewhat vague but very general. Equating “singleton” with “god” is unfair IMO. I quote nick:

    A democratic world republic could be a kind of singleton, as could a world dictatorship. A friendly superintelligent machine could be another kind of singleton, assuming it was powerful enough that no other entity could threaten its existence or thwart its plans. A “transcending upload” that achieves world domination would be another example. Yet another way in which a singleton could form is through convergent evolution, e.g. if it turns out that all sufficiently advanced individuals or cultures come to accept fundamentally the same values or goals. These common values in combination with all the individuals and cultures that embrace them would then be an “agency” in the broad sense intended here, and it would constitute a singleton. One could also imagine a singleton arising from the universal spread of a single self-enforcing moral code. … A singleton need not be monolithic. It could contain within itself an enormous variety of independent agents each pursuing their own disparate goals, just as is the case in a liberal democratic state. The goals and actions of the singleton could be decided by its inhabitants or their elected representatives. … The concept of a singleton is thus much broader and more abstract than concept of a world government. A world government (in the ordinary sense of the word) is only one type of singleton among many.

    Nevertheless, all the possible singletons share one essential property: the ability to solve global coordination problems.

  • http://profile.typekey.com/sentience/ Eliezer Yudkowsky

    @Robin: Well, either you and I have really different visualizations of what the coherent parts of humanity’s reflective equilibria would look like, or you don’t think the Friendly AI project has the described outcome, or you have a really different moral reaction to that outcome.

    If an AI goes FOOM, you seem to recognize that condition, or that prospect, as “total war”. Afterward, you seem to recognize the resultant as a “God”, and its relation to humanity as “rule”. So either we’ve got really different visualizations of this process, or we have really different moral reactions to it. This seems worth exploring, because I suspect that it accounts for a large fraction of the real fuel in the argument.

  • James Andrix

    Tim:
    A Mature FAI could modify its own utility function, if it wanted to. If it had good reasons that you and I would find compelling. There is no lock-in other than doing what we would want and not what we wouldn’t want.

    Robin:
    I can naively imagine many post singularity scenarios with no FAI that I would want to live in. I didn’t understand that the many of the high probability scenarios were so awful until you explained it to me.

    Extinction is just one of the awful things the singularity could bring.

    I’m all for uploading, nanotech, and and agressive colonization wave, but what’s the benefit of it all? I previously view these things as potentially dangerous but overall good things to have around. Now I come across an argument that the dangers I supposed are the most likely primary effects!

    This SEEMS to me like values argument, that you are ok with the described slavery and murder. You’ve disclaimed endorsement of the future you describe, and I’m giving you credit for not being a murderous slaver… But you also don’t seem to want to do anything to alter this future.

    A strategy for changing this future doesn’t need to take the form of a singleton. It CAN’T rely on a singleton, because part of the argument is that singletons aren’t likely. God is not an option, so we have to use evolution. (or somethng else)

    I think it’s a fairly common goal among singularitarians that no sentient being will have to do anything they don’t want to do. We enjoy life, and the nanites do all the work. That is what I see as the benefit to these technologies, that is what they are FOR. To serve us, and our goals. Going from that to a future where people get deleted for being uneconomical is losing a LOT. Getting a better future is well worth a slightly delayed colonization wave.

  • Douglas Knight

    If you assume perfectly rational agents, then they will optimize over the whole future and it will be as if there is only one round of selection, so it will select for agents trying to maximize expected population. Such agents will recognize the main bottleneck and be risk-neutral about it. At some point, we will enter that regime, unless there is insufficient diversity to allow that outcome; a founder effect is much like a singleton.

  • Tim Tyler

    If you assume perfectly rational agents, then they will optimize over the whole future and it will be as if there is only one round of selection, so it will select for agents trying to maximize expected population. Such agents will recognize the main bottleneck and be risk-neutral about it.

    What state a superintelligent agent will aim at – and its level of risk aversion – will be properties of its utility function. You can’t easily generalise over all such agents – because they could be very different from one another.

  • http://hanson.gmu.edu Robin Hanson

    Douglas, as I told Carl, you can’t just assume you know that maximizing expected population is “perfectly rational” choice. I’ve argued evolution selects for maxing expected log (quality-weighted) population (when that is the only long term resource).

  • michael vassar

    Robin: You CAN assume that you will find the largest populations are the populations of those rational agents that maximize population, or roughly equivalently, that most resources are held by those rational agents which maximize resources under their control. Non-myopic agents don’t need to let evolution sculpt their utility functions. They can adjust their behavior to evolutionary realities simply by understanding evolution

    Seriously Robin, just think about the situation described very carefully. It’s a math problem. There’s only one right answer. Douglas Knight, Carl, Eliezer and I are all very reliable sources for correct answers on published math problems, including answers on whether the word problem translates into the math you are claiming it does.

  • http://hanson.gmu.edu Robin Hanson

    Michael, Above, Eliezer seems to accept my math claim. I don’t yet know if Carl is convinced or not. My disagreement with Douglas was about what “rational” means, which is not a math question. I certainly agree that all else equal you can expect to see more X in creatures that max X, vs. creatures that max log(X), but that is a different question from what preferences evolution selects for.

  • michael vassar

    Yes, but it’s the question we need to ask in order to understand what the hardscrapple frontier would probably look like.

  • http://profile.typekey.com/sentience/ Eliezer Yudkowsky

    I don’t consider myself a superreliable math source. If the fate of the world isn’t at stake, I’ll often state an intuition rather than trying to prove it. For that matter, if the fate of the world were at stake, the first thing I’d do would be consult Marcello.

    Robin, I accept the part about locally logarithmic behavior on spatially global and temporally local problems when there will be many future options and all are multiplicative. I don’t accept the claim that evolution turns future entities into log-population maximizers. In a sense, you’ve actually shown just the opposite; because aggregative maximizers or log-maximizers will both show instrumental log-seeking behavior, entities with terminal log valuations have no fitness advantage. Evolution requires visible differences of behavior on which to operate.

    If there are many non-multiplicative options – say, there are ways to form trustworthy contracts, and a small party can contract with an intergalactic Warren Buffett “I will give you 10% of my lineage’s resources now, if you agree to use the same amount of resources to recreate copies of me in a billion years,” then it’s not clear to me that logarithmics have an advantage; most of the numbers might be in aggregators because numbers are what they want, and that’s what they use non-multiplicative options to get.

  • http://hanson.gmu.edu Robin Hanson

    Eliezer, I agree one might analyze non-multiplicative worlds, but no one has done so yet, and the world so far has been pretty multiplicative. Please recall that I was initially responding to confident claims by Carl and others that evolution would make for terrible wars over the solar system because evolved creatures would be terminal-outcome-oriented and risk-neutral about such outcomes. In this context I make three claims:

    1. It is not obvious evolution would create terminal-outcome-oriented creatures,
    2. It is not obvious such creatures would be risk-neutral about terminal outcomes, and
    3. Even if they were they would have to be rather confident this conflict was in fact the last such conflict to be risk-neutral about resources gained from it.

    Do you disagree with any of these claims?

  • Douglas Knight

    I used “rational” to mean not myopic. You can just assume incorrect usage, but you can’t make me like it.

    Evolution of myopic agents often produces ones that maximize log-resources. Evolution of non-myopic agents produces ones that maximize resources.

    The question is: will there be enough non-myopia and diversity among Ems that the second effect kicks in? I expect evolution of Ems to head towards non-myopia, so the question is only about diversity, after the escape from myopia.

    (By “evolution” I mean sampling afterwards by resource weighting.)

  • http://hanson.gmu.edu Robin Hanson

    Douglas, you seem to just ignore the math I’m presenting and citing; do you know of some proof that makes it just completely irrelevant? Do you know of any formal analysis of the selection of non-myopic agents that supports your claim?

  • Douglas Knight

    The work you cite assumes myopia. You seem to just ignore that.

    The non-myopic case is very simple, at least assuming zero-sum resources (by which I mean the thing that we use to weight the final observation), which I should have said. The key point is “they will optimize over the whole future and it will be as if there is only one round of selection,” or, as other people put it, only terminal values matter. To analyze the case of one zero-sum round of selection, there is nothing more to say than “linearity of expectation.”

  • http://profile.typekey.com/sentience/ Eliezer Yudkowsky

    I don’t know about evolution creating terminal-outcome-oriented creatures, but the case for self-modifying AIs by default converging to expected utility maximization has been written up by e.g. Omohundro. But I think that what you mean here is aggregate valuation by expected utility maximizers. This wouldn’t be created per se by either evolution or self-modification, but it also seems fairly likely to emerge as an idiom among utility functions not strictly specified. Other possible minds could be satisficers, and these would be less of a threat in a competitive situation (they would only take over the world if they knew they could win, or if they expected a strong threat to their button-to-keep-pressed if they weren’t in sole charge of the galaxy).

  • http://hanson.gmu.edu Robin Hanson

    I’m frustrated that I seem unable communicate what should be a precise technical claim: evolution need not select for creatures who maximize expected future descendants. People keep claiming this as if it had been proven, but it has not, because it is not so.

    The paper I cite is a clear precise counter-example. It considers a case where choices and probabilities are independent across time periods, and in this case its is optimal, non-myopically, to make choices locally in time to max the expected log of period payoffs.

    That case easily generalizes to chunks of N periods that are correlated arbitrarily internally, but independent across chunks. Again agents max the expected sum of log period returns, which is the same as maxing the expected sum of chunk returns. And you can make N as large as you like.

  • Douglas Knight

    The simplest example demonstrating the same thing is the environment in which actions have no consequences. In that environment, there is no differential selection between expectation maximizers and expected log maximizers. cf Eliezer

  • James Andrix

    Robin:
    I think part of the problem may be confusion of future descendants with future control of resources.

    Robin-The paper I cite is a clear precise counter-example. It considers a case where choices and probabilities are independent across time periods, and in this case its is optimal, non-myopically, to make choices locally in time to max the expected log of period payoffs.

    Douglas-Evolution of non-myopic agents produces ones that maximize resources.

    What I get from Douglas is that every ‘generation’of an EM or AI can plan ahead the whole future and behave optimally (for resources) in each situation. (or create offspring that behave optimally)

    What I get from you is that creatures that have evolved logarithmic preferences do the best, in resources.

    I think he’s trying to answer the question:
    “Even if you somehow knew that it was in fact that last extinction possibility forevermore, how could evolutionary selection have favored a different attitude toward such that event?”
    with ‘By making things smart enough to do the math with a linear utility function.’

    And I think you’re answering: ‘They can but that’s a losing option.’

    You seem to agree that Evolution will create resource grabbers. It’s tautological that the universe will go to those who are best at getting the universe, whatever that means. (But I bet it’s not meekness)

  • Tim Tyler

    Non-myopic agents can indeed ration their resources and avoid short term reproduction for the sake of their own long term success. I’m not sure if this is what you are asking for – but it seems like a pretty trivial observation to me: they form a government, and the government imposes rationing. We have seen plenty of historical examples of such rationing.

    Reproductive success is indeed not evolution’s utility function – see the work of Dewar – or my essay: God’s utility function.

  • http://www.weidai.com Wei Dai

    What does “select for” mean? In Sinn’s paper, the population with the goal of maximizing expected future descendants do end up having a larger number of expected future descendants, compared with the population that wants to maximize log descendants. But Robin concludes that the former is not “selected for” because if we look at the actual probability distribution of future descendants, most of their descendants are concentrated into a few possible worlds. The log maximizers, while having fewer expected descendants, have them spread more evenly over possible worlds. So we end up with the following situation:

    1. If we randomly sample a possible world and then randomly sample a creature in that world, then with high probability (approaching 1 as time goes to infinity) we’ll find an expected-log-population maximizer.
    2. If we randomly sample a creature across all possible worlds, then with high probability (approaching 1 as time goes to infinity) we’ll find an expected-population maximizer.

    Does it make sense to say that log maximizers are selected for, based on 1, when 2 is also true? For someone who believes the MWI, I think the answer should be no, so I’m surprised that Robin accepts Sinn’s interpretation of the math. I’d be interested in an explanation of why given MWI, the much higher population of expected-population maximizer in a few branches doesn’t make up for the somewhat lower population in most branches.

  • http://hanson.gmu.edu Robin Hanson

    Douglas, a valid example.

    Wei, that seems worth pondering.

  • http://reflectivedisequilibria.blogspot.com/ Carl Shulman

    “Wei, that seems worth pondering.”

    It wasn’t when I made the same point?

  • Douglas Knight

    Wei,
    I assumed zero-sum games to avoid this issue, or a similar one.

    However you sample, you’ll select for agents whose utility function is closest to the probability of being sampled.

  • http://hanson.gmu.edu Robin Hanson

    Wei, does it make a difference if we change the model, dividing the previous amounts by some finite resource pot (perhaps exponentially growing) that is divided among the lineages? That should change the relative fraction across worlds of those linear-value creature who dominate in a few rare worlds.

  • http://www.weidai.com Wei Dai

    Robin, if we allow any kind of interaction between the choices of the two lineages, then it becomes a two-player game, and I don’t know if we can still prove anything general like Sinn did. However it seems safe to say that the linear-value creatures will still achieve a higher expected population in the long run, and the log-value creatures will still achieve a higher expected log population in the long run. And statements 1 and 2 in my comment above still follow.

    Here’s a simple example to get a sense of what might happen. Suppose the universe may be type A, in which case it supports at most 10^100 creatures, or type B, in which case it supports up to 10^200 creatures. At time 0 everyone believes the universe is type A with probability 0.9999 and type B with probability 0.0001. Our two types of creatures fight a war at time 0, and they have two strategies to select from. If both sides choose the same strategy, they reach a stalemate, otherwise whoever chooses strategy A wins if the universe turns out to be type A, and whoever chooses strategy B wins if the universe turns out to be type B. It should be obvious that the outcome will be that the linear-value creatures choose strategy B, and the log-value creatures choose strategy A, and they each get what they want, respectively higher expected population and higher expected log population.

    Douglas, I don’t see how you can assume zero-sum in this case. We have one type of creature that values expected population, and another that values expected log population. Unless you have population1 + log(population2) = constant, which makes no sense as a realistic constraint, you can’t have a zero-sum game.

    Carl, I’m guessing it’s the MWI implications that made Robin rethink.

  • http://profile.typepad.com/halfinney Hal Finney

    Wei, going back to your 1st comment, does the fact that we are log maximizers (supposing that we are, at least we seem to be risk averse while I gather that linear maximizers would be risk neutral) then tell us something about the nature of the MWI and how anthropic selection works? Wouldn’t we expect to find ourselves as linear maximizers, according to some models of the anthropic principle?

  • http://www.weidai.com Wei Dai

    Hal, I don’t think we actually are log maximizers. We are clearly risk averse with respect to physical resources or wealth in general, but that’s because there’s diminishing return in converting wealth into descendants. (Note that males are much less risk averse than females because males are subject to less diminishing return.) I don’t think we are risk averse with respect to the number of descendants, but I don’t know where to find the evidence to show that conclusively. One complication is that evolution didn’t program us to deliberately maximize either expected descendants or expected log descendants, so we have to determine whether in our environment of evolutionary adaptation we would have had the effect of maximizing expected descendants or expected log descendants. This seems non-trivial but maybe someone can think of a clever way?

    Your observation that whether we are log maximizers or linear maximizers might tell us something about the nature of MWI and anthropic selection seems like a really good one. Almost too good to be true, so I’ll have to think it over a bit…

  • Douglas Knight

    I meant zero-sum with respect to the weighting on how you sample.
    Maybe I shouldn’t have used that term.

    A similar scenario that doesn’t select for expectation maximizers is the scenario in which they don’t exist.

  • Will Pearson

    linear-value creature who dominate in a few rare worlds.

    Robin, rare enough to to be mangled in theory?

  • http://www.weidai.com Wei Dai

    Having had time to think about Sinn’s paper some more, it now seems that Sinn’s risk model is less relevant than I first realized. Sinn’s prediction (in the sense of statement 1 in my first comment) of risk aversion with regard to the number of descendants only applies to risk that remains perfectly correlated regardless of the size of one’s lineage, but one benefit of having more descendants is in fact greater diversification against almost all kinds of risk: you can send your descendants to different corners of the earth or instruct them to exploit different local niches. The only kind of risk in our environment of evolutionary adaptation (EEA) I can think of that can’t be hedged against by having more descendants is something like a supernova or asteroid strike large enough to cause global extinction, for which being risk averse wouldn’t have made any difference anyway.

    As I tried to think of ways to test Sinn’s theory, I kept noticing that any candidate test I came up with couldn’t either confirm or disprove his theory no matter what the data says because it didn’t satisfy his risk model. It seems that Sinn never considered how his theory might be tested, or I think he would have realized how unrealistic his risk model is.

    Robin wrote: So they can be terrified of global catastrophe, mildly concerned about car accidents, and completely indifferent to within-lineage tournaments.

    No, if we change Sinn’s model even slightly, then his prediction breaks down completely. Let’s say that risks within populations are perfectly correlated, but when a population gets to a certain size, it splits into two subpopulations such that risks are perfectly correlated within each subpopulation, but independent between subpopulations. This describes car accidents. It’s obvious that such groups of populations are essentially equivalent to groups of individual reproducing organisms with uncorrelated risk, so there is no reason to be risk averse.

    Hal, unfortunately the specific suggestion you made is too good to be true, but it might be worth think about what other predictions would differ between the two sampling methods, and see if any of them can be tested. That’s not quite on topic so if you have any ideas, we should probably discuss them elsewhere.

  • Will Pearson

    I’d be interested in continuing the discussion, elsewhere. Although we don’t have anywhere else at the moment, here will do for now.

    One possible risk associated with being too numerous is the risk from parasites. It is the most convincing evolutionary reason I have heard for sexuality, which halves the lineage of the reproducers genes. Some background.

    Whether such risk will exist in the future is another matter.

  • http://www.weidai.com Wei Dai

    Will, sex/parasites is not a good test, because at high population densities parasites are inevitable, and therefore even risk-neutral linear-value creatures will adopt costly defenses like sex (plus I think sexual reproduction has other benefits like allowing a bigger genome to be maintained against mutations so they might adopt sex even without parasites). In other words, sexual reproduction increases expected descendants, not just expected log descendants.

    And also, the only way we could have evolved a general cognitive adaption of risk aversion with respect to the number of descendants (which is what Robin cited Sinn’s paper for) as opposed to specific, one-off mechanisms like sex, is if there were many different kinds of correlated risk that couldn’t be hedged against by maximizing expected descendants and naturally breaking them off into uncorrelated or less correlated subpopulations, but could be hedged some other way which involves a smaller expected number of descendants. That just doesn’t seem plausible, given that it’s hard to think of even one example.

  • http://hanson.gmu.edu Robin Hanson

    Did anyone look at my math in the post? I think it shows pretty robustly that most any utility over ultimate gains translates to near log utility over each short period relative returns, when there are many not highly correlated periods. The issue isn’t whether long term utility is log or linear, the issue is that as long as the future will be long, you are forced to have log utility over short term gains.

  • http://www.weidai.com Wei Dai

    Robin, I was trying to understand Sinn’s math, and didn’t noticed that you’ve introduced novel math of your own. Looking at it now, I can’t tell how you derived the conclusion “a wide range of U(x), u(x) goes to log(x) for N large” so I don’t know if that is correct or not(*). But from what I can tell, you make the same unrealistic assumption that Sinn did, namely that risks are perfectly correlated within a population no matter how large it is (whereas common sense says that the larger the population, the less correlated the risks its members face), and populations can’t split into independent subpopulations. The key phrase here is “if probabilities and choices are independent across timesteps”. What happens in your model if whenever a population reaches some threshold size, it splits into two subpopulations each of which then faces an independent set of choices?

    (*) Suppose U(x)=x, then doubling r_0 doubles U(prod r_t) but only increases sum u(r_t) by some amount independent of r_1, r_2, …, etc. So how could U(prod r_t)=sum u(r_t)? You’re also taking the expectation value somewhere right, and just omitting it from your notation? There’s too many missing pieces for me to figure out what you’re doing…

  • Will Pearson

    Wei, my point was not to say that we are log utility maximisers because of sex, but that parasitism is a risk that increases in proportion (or some non-linear function) with your lineage (assuming it is dense and connected). Diseases and parasites spread more quickly in a dense than sparse, monocultures bad etc.

    Look at sex from the genes point of view not the individuals. It is halving its lineage by adopting sexuality rather than asexuality. A genome having two offspring could have two of 100% its own kids, rather than effectively fostering a random strangers kid. If that is an example of the strategies that an evolutionary lineage maximizing AI adopts, then population of AIs may decide to keep humans around because we are not affected by the same parasitic memes.

    We can effectively tell nothing about what an AI trying maximize it’s lineage (log or otherwise) will do because we don’t know what troubles it will face, because we don’t know how it will work.

  • http://www.weidai.com Wei Dai

    Will, thanks for the clarification. I think I understand your point now.

    Robin, have you had a chance to consider my comments on your and Sinn’s math? Whether evolution tends to produce linear-value creatures or log-value creatures (under assumptions that approximate our own past) seems like an interesting and solvable problem, and I hope we can reach some resolution instead of just letting the discussion trail off.

  • http://profile.typepad.com/robinhanson Robin Hanson

    Wei, the literature distinguishes correlated and uncorrelated risks, and everyone agrees that near risk-neutrality is appropriate for uncorrelated risks. In the world of finance there are correlated risks, and this literature has been interested in selection of investment strategies in finance. So it is natural and appropriate to focus on modeling the selection of correlated risks.

  • http://www.weidai.com Wei Dai

    Sorry for the late reply. I don’t know how to keep track of responses, other than to revisit old posts when I can remember to.

    Robin, you did not object to my characterization of your math as assuming perfectly correlated risks regardless of population size, so I’ll assume that is correct. You also didn’t try to argue that the assumption is realistic, or that your conclusions are insensitive to the assumption, even though I offered reasons why I suspect both might be false.

    I don’t know why you talked about the difference between correlated and uncorrelated risks, or the fact that correlated risks exist and is studied in the world of finance. Did I somehow indicate a failure to understand the difference between correlated and uncorrelated risks, or said that I object to any assumption of correlated risks, as opposed to perfectly correlated risks regardless of population size?

  • http://profile.typepad.com/robinhanson Robin Hanson

    Wei, one can decompose arbitrary risks into correlated and uncorrelated risks, and preferences can treat those components differently. Since it seems clear how preferences treat the uncorrelated part, the issue is how it treats the correlated part. For the purpose of studying that question it is as I said natural and appropriate to study a model of fully correlated risks.

  • http://www.weidai.com Wei Dai

    Thanks, Robin. I think I now understand your point. But putting it aside for a moment (I’ll come back to it), it looks to me that the mathematical reasoning in your post just isn’t right, and that your conclusions don’t follow from your assumptions. Let’s consider a specific numerical example, with an asexual species and U(x)=x. Say the number of periods is 3000, and in each period the choices are R (risky) and C (conservative). If a creature chooses C, he is survived by 100 offspring. If a creature chooses R, he has 50/50 chance of either 1 offspring, or 1000 offspring. The risks are fully correlated within a period, so everyone who chooses R has the same number of offspring. Probabilities are independent across periods. This example satisfies your assumptions, right?

    If u(x)=log(x), then creatures should choose C every period, since log(100)=2 > log(1)/2+log(1000)/2=1.5. But choosing R every period maximizes the expected population at the end of 3000 periods. To see this, the expected population of always choosing R is at least .5^3000 * 1000^3000 = 500^3000, which is the probability that r_t=1000 for 3000 periods, times the total population if that were to occur. Choosing C leads to a population of 100^3000 with probability 1, less than the expected population of choosing R. It seems clear that u(x) does not go to log(x) if U(x)=x.

    Robin, can you check if my analysis is correct?

    But either way, Sinn’s math still stands, so let’s go back to the question of whether modeling only fully correlated risks makes sense. First, we can check that Sinn’s conclusions do apply in the example above: choosing R leads to a greater expected population, but with high probability the actual population will be less than choosing C. So it seems that evolution selects for u(x)=log(x) if we defined “select for” as Sinn’s “evolutionary dominance” (ignoring MWI considerations for the moment). But what if the environment also has uncorrelated risks? Suppose that odd periods stay the same but in even periods, the risks of choosing R are completely uncorrelated. Then evolution should select for creatures with time-dependent utility: u(x,t)={x if t is even, log(x) if t is odd}.

    In real life correlations of risks do not change this predictably with time, so under Sinn’s formalism, evolution should select for creatures with dynamic utility functions that change depending on the creature’s estimate of the degree of correlation of the risk in the decision he faces. But that abuses the concept of utility function beyond recognition. Consider the analogy with the theory of investments, where there aren’t utility functions over the outcomes of individual investments (changing depending on their risk characteristics). Instead one has an utility function over one’s income stream, and risk aversion or neutrality on individual investments emerge from selecting strategies to maximize expected utility under that fixed utility function.

    So, I think it makes more sense to say that evolution also selects for behavioral strategies, not utility functions. These strategies tended to maximize expected descendants when risks were uncorrelated and expected log descendants when risks were correlated. That fits better anyway with the idea that we are adaptation executors, not utility maximizers, and perhaps explains why we don’t seem to have direct preferences over the number of our descendants.

  • Pingback: Overcoming Bias : Dreamtime Finance

  • Pingback: AI Foom Debate: Post 35 – 40 | wallowinmaya