To a first approximation, the future will either be a singleton, a single integrated power choosing the future of everything, or it will be competitive, with conflicting powers each choosing how to perpetuate themselves. Selection effects apply robustly to competition scenarios; some perpetuation strategies will tend to dominate the future. To help us choose between a singleton and competition, and between competitive variations, we can analyze selection effects to understand competitive scenarios. In particular, selection effects can tell us the key feature without which it is very hard to forecast:
Thanks, Robin. I think I now understand your point. But putting it aside for a moment (I'll come back to it), it looks to me that the mathematical reasoning in your post just isn't right, and that your conclusions don't follow from your assumptions. Let's consider a specific numerical example, with an asexual species and U(x)=x. Say the number of periods is 3000, and in each period the choices are R (risky) and C (conservative). If a creature chooses C, he is survived by 100 offspring. If a creature chooses R, he has 50/50 chance of either 1 offspring, or 1000 offspring. The risks are fully correlated within a period, so everyone who chooses R has the same number of offspring. Probabilities are independent across periods. This example satisfies your assumptions, right?
If u(x)=log(x), then creatures should choose C every period, since log(100)=2 > log(1)/2+log(1000)/2=1.5. But choosing R every period maximizes the expected population at the end of 3000 periods. To see this, the expected population of always choosing R is at least .5^3000 * 1000^3000 = 500^3000, which is the probability that r_t=1000 for 3000 periods, times the total population if that were to occur. Choosing C leads to a population of 100^3000 with probability 1, less than the expected population of choosing R. It seems clear that u(x) does not go to log(x) if U(x)=x.
Robin, can you check if my analysis is correct?
But either way, Sinn's math still stands, so let's go back to the question of whether modeling only fully correlated risks makes sense. First, we can check that Sinn's conclusions do apply in the example above: choosing R leads to a greater expected population, but with high probability the actual population will be less than choosing C. So it seems that evolution selects for u(x)=log(x) if we defined "select for" as Sinn's "evolutionary dominance" (ignoring MWI considerations for the moment). But what if the environment also has uncorrelated risks? Suppose that odd periods stay the same but in even periods, the risks of choosing R are completely uncorrelated. Then evolution should select for creatures with time-dependent utility: u(x,t)={x if t is even, log(x) if t is odd}.
In real life correlations of risks do not change this predictably with time, so under Sinn's formalism, evolution should select for creatures with dynamic utility functions that change depending on the creature's estimate of the degree of correlation of the risk in the decision he faces. But that abuses the concept of utility function beyond recognition. Consider the analogy with the theory of investments, where there aren't utility functions over the outcomes of individual investments (changing depending on their risk characteristics). Instead one has an utility function over one's income stream, and risk aversion or neutrality on individual investments emerge from selecting strategies to maximize expected utility under that fixed utility function.
So, I think it makes more sense to say that evolution also selects for behavioral strategies, not utility functions. These strategies tended to maximize expected descendants when risks were uncorrelated and expected log descendants when risks were correlated. That fits better anyway with the idea that we are adaptation executors, not utility maximizers, and perhaps explains why we don't seem to have direct preferences over the number of our descendants.
Wei, one can decompose arbitrary risks into correlated and uncorrelated risks, and preferences can treat those components differently. Since it seems clear how preferences treat the uncorrelated part, the issue is how it treats the correlated part. For the purpose of studying that question it is as I said natural and appropriate to study a model of fully correlated risks.
Sorry for the late reply. I don't know how to keep track of responses, other than to revisit old posts when I can remember to.
Robin, you did not object to my characterization of your math as assuming perfectly correlated risks regardless of population size, so I'll assume that is correct. You also didn't try to argue that the assumption is realistic, or that your conclusions are insensitive to the assumption, even though I offered reasons why I suspect both might be false.
I don't know why you talked about the difference between correlated and uncorrelated risks, or the fact that correlated risks exist and is studied in the world of finance. Did I somehow indicate a failure to understand the difference between correlated and uncorrelated risks, or said that I object to any assumption of correlated risks, as opposed to perfectly correlated risks regardless of population size?
Wei, the literature distinguishes correlated and uncorrelated risks, and everyone agrees that near risk-neutrality is appropriate for uncorrelated risks. In the world of finance there are correlated risks, and this literature has been interested in selection of investment strategies in finance. So it is natural and appropriate to focus on modeling the selection of correlated risks.
Will, thanks for the clarification. I think I understand your point now.
Robin, have you had a chance to consider my comments on your and Sinn's math? Whether evolution tends to produce linear-value creatures or log-value creatures (under assumptions that approximate our own past) seems like an interesting and solvable problem, and I hope we can reach some resolution instead of just letting the discussion trail off.
Wei, my point was not to say that we are log utility maximisers because of sex, but that parasitism is a risk that increases in proportion (or some non-linear function) with your lineage (assuming it is dense and connected). Diseases and parasites spread more quickly in a dense than sparse, monocultures bad etc.
Look at sex from the genes point of view not the individuals. It is halving its lineage by adopting sexuality rather than asexuality. A genome having two offspring could have two of 100% its own kids, rather than effectively fostering a random strangers kid. If that is an example of the strategies that an evolutionary lineage maximizing AI adopts, then population of AIs may decide to keep humans around because we are not affected by the same parasitic memes.
We can effectively tell nothing about what an AI trying maximize it's lineage (log or otherwise) will do because we don't know what troubles it will face, because we don't know how it will work.
Robin, I was trying to understand Sinn's math, and didn't noticed that you've introduced novel math of your own. Looking at it now, I can't tell how you derived the conclusion "a wide range of U(x), u(x) goes to log(x) for N large" so I don't know if that is correct or not(*). But from what I can tell, you make the same unrealistic assumption that Sinn did, namely that risks are perfectly correlated within a population no matter how large it is (whereas common sense says that the larger the population, the less correlated the risks its members face), and populations can't split into independent subpopulations. The key phrase here is "if probabilities and choices are independent across timesteps". What happens in your model if whenever a population reaches some threshold size, it splits into two subpopulations each of which then faces an independent set of choices?
(*) Suppose U(x)=x, then doubling r_0 doubles U(prod r_t) but only increases sum u(r_t) by some amount independent of r_1, r_2, ..., etc. So how could U(prod r_t)=sum u(r_t)? You're also taking the expectation value somewhere right, and just omitting it from your notation? There's too many missing pieces for me to figure out what you're doing...
Did anyone look at my math in the post? I think it shows pretty robustly that most any utility over ultimate gains translates to near log utility over each short period relative returns, when there are many not highly correlated periods. The issue isn't whether long term utility is log or linear, the issue is that as long as the future will be long, you are forced to have log utility over short term gains.
Will, sex/parasites is not a good test, because at high population densities parasites are inevitable, and therefore even risk-neutral linear-value creatures will adopt costly defenses like sex (plus I think sexual reproduction has other benefits like allowing a bigger genome to be maintained against mutations so they might adopt sex even without parasites). In other words, sexual reproduction increases expected descendants, not just expected log descendants.
And also, the only way we could have evolved a general cognitive adaption of risk aversion with respect to the number of descendants (which is what Robin cited Sinn's paper for) as opposed to specific, one-off mechanisms like sex, is if there were many different kinds of correlated risk that couldn't be hedged against by maximizing expected descendants and naturally breaking them off into uncorrelated or less correlated subpopulations, but could be hedged some other way which involves a smaller expected number of descendants. That just doesn't seem plausible, given that it's hard to think of even one example.
I'd be interested in continuing the discussion, elsewhere. Although we don't have anywhere else at the moment, here will do for now.
One possible risk associated with being too numerous is the risk from parasites. It is the most convincing evolutionary reason I have heard for sexuality, which halves the lineage of the reproducers genes. Some background.
Whether such risk will exist in the future is another matter.
Having had time to think about Sinn's paper some more, it now seems that Sinn's risk model is less relevant than I first realized. Sinn's prediction (in the sense of statement 1 in my first comment) of risk aversion with regard to the number of descendants only applies to risk that remains perfectly correlated regardless of the size of one's lineage, but one benefit of having more descendants is in fact greater diversification against almost all kinds of risk: you can send your descendants to different corners of the earth or instruct them to exploit different local niches. The only kind of risk in our environment of evolutionary adaptation (EEA) I can think of that can't be hedged against by having more descendants is something like a supernova or asteroid strike large enough to cause global extinction, for which being risk averse wouldn't have made any difference anyway.
As I tried to think of ways to test Sinn's theory, I kept noticing that any candidate test I came up with couldn't either confirm or disprove his theory no matter what the data says because it didn't satisfy his risk model. It seems that Sinn never considered how his theory might be tested, or I think he would have realized how unrealistic his risk model is.
Robin wrote: So they can be terrified of global catastrophe, mildly concerned about car accidents, and completely indifferent to within-lineage tournaments.
No, if we change Sinn's model even slightly, then his prediction breaks down completely. Let's say that risks within populations are perfectly correlated, but when a population gets to a certain size, it splits into two subpopulations such that risks are perfectly correlated within each subpopulation, but independent between subpopulations. This describes car accidents. It's obvious that such groups of populations are essentially equivalent to groups of individual reproducing organisms with uncorrelated risk, so there is no reason to be risk averse.
Hal, unfortunately the specific suggestion you made is too good to be true, but it might be worth think about what other predictions would differ between the two sampling methods, and see if any of them can be tested. That's not quite on topic so if you have any ideas, we should probably discuss them elsewhere.
Hal, I don't think we actually are log maximizers. We are clearly risk averse with respect to physical resources or wealth in general, but that's because there's diminishing return in converting wealth into descendants. (Note that males are much less risk averse than females because males are subject to less diminishing return.) I don't think we are risk averse with respect to the number of descendants, but I don't know where to find the evidence to show that conclusively. One complication is that evolution didn't program us to deliberately maximize either expected descendants or expected log descendants, so we have to determine whether in our environment of evolutionary adaptation we would have had the effect of maximizing expected descendants or expected log descendants. This seems non-trivial but maybe someone can think of a clever way?
Your observation that whether we are log maximizers or linear maximizers might tell us something about the nature of MWI and anthropic selection seems like a really good one. Almost too good to be true, so I'll have to think it over a bit...
Wei, going back to your 1st comment, does the fact that we are log maximizers (supposing that we are, at least we seem to be risk averse while I gather that linear maximizers would be risk neutral) then tell us something about the nature of the MWI and how anthropic selection works? Wouldn't we expect to find ourselves as linear maximizers, according to some models of the anthropic principle?
Robin, if we allow any kind of interaction between the choices of the two lineages, then it becomes a two-player game, and I don't know if we can still prove anything general like Sinn did. However it seems safe to say that the linear-value creatures will still achieve a higher expected population in the long run, and the log-value creatures will still achieve a higher expected log population in the long run. And statements 1 and 2 in my comment above still follow.
Here's a simple example to get a sense of what might happen. Suppose the universe may be type A, in which case it supports at most 10^100 creatures, or type B, in which case it supports up to 10^200 creatures. At time 0 everyone believes the universe is type A with probability 0.9999 and type B with probability 0.0001. Our two types of creatures fight a war at time 0, and they have two strategies to select from. If both sides choose the same strategy, they reach a stalemate, otherwise whoever chooses strategy A wins if the universe turns out to be type A, and whoever chooses strategy B wins if the universe turns out to be type B. It should be obvious that the outcome will be that the linear-value creatures choose strategy B, and the log-value creatures choose strategy A, and they each get what they want, respectively higher expected population and higher expected log population.
Douglas, I don't see how you can assume zero-sum in this case. We have one type of creature that values expected population, and another that values expected log population. Unless you have population1 + log(population2) = constant, which makes no sense as a realistic constraint, you can't have a zero-sum game.
Carl, I'm guessing it's the MWI implications that made Robin rethink.
Thanks, Robin. I think I now understand your point. But putting it aside for a moment (I'll come back to it), it looks to me that the mathematical reasoning in your post just isn't right, and that your conclusions don't follow from your assumptions. Let's consider a specific numerical example, with an asexual species and U(x)=x. Say the number of periods is 3000, and in each period the choices are R (risky) and C (conservative). If a creature chooses C, he is survived by 100 offspring. If a creature chooses R, he has 50/50 chance of either 1 offspring, or 1000 offspring. The risks are fully correlated within a period, so everyone who chooses R has the same number of offspring. Probabilities are independent across periods. This example satisfies your assumptions, right?
If u(x)=log(x), then creatures should choose C every period, since log(100)=2 > log(1)/2+log(1000)/2=1.5. But choosing R every period maximizes the expected population at the end of 3000 periods. To see this, the expected population of always choosing R is at least .5^3000 * 1000^3000 = 500^3000, which is the probability that r_t=1000 for 3000 periods, times the total population if that were to occur. Choosing C leads to a population of 100^3000 with probability 1, less than the expected population of choosing R. It seems clear that u(x) does not go to log(x) if U(x)=x.
Robin, can you check if my analysis is correct?
But either way, Sinn's math still stands, so let's go back to the question of whether modeling only fully correlated risks makes sense. First, we can check that Sinn's conclusions do apply in the example above: choosing R leads to a greater expected population, but with high probability the actual population will be less than choosing C. So it seems that evolution selects for u(x)=log(x) if we defined "select for" as Sinn's "evolutionary dominance" (ignoring MWI considerations for the moment). But what if the environment also has uncorrelated risks? Suppose that odd periods stay the same but in even periods, the risks of choosing R are completely uncorrelated. Then evolution should select for creatures with time-dependent utility: u(x,t)={x if t is even, log(x) if t is odd}.
In real life correlations of risks do not change this predictably with time, so under Sinn's formalism, evolution should select for creatures with dynamic utility functions that change depending on the creature's estimate of the degree of correlation of the risk in the decision he faces. But that abuses the concept of utility function beyond recognition. Consider the analogy with the theory of investments, where there aren't utility functions over the outcomes of individual investments (changing depending on their risk characteristics). Instead one has an utility function over one's income stream, and risk aversion or neutrality on individual investments emerge from selecting strategies to maximize expected utility under that fixed utility function.
So, I think it makes more sense to say that evolution also selects for behavioral strategies, not utility functions. These strategies tended to maximize expected descendants when risks were uncorrelated and expected log descendants when risks were correlated. That fits better anyway with the idea that we are adaptation executors, not utility maximizers, and perhaps explains why we don't seem to have direct preferences over the number of our descendants.
Wei, one can decompose arbitrary risks into correlated and uncorrelated risks, and preferences can treat those components differently. Since it seems clear how preferences treat the uncorrelated part, the issue is how it treats the correlated part. For the purpose of studying that question it is as I said natural and appropriate to study a model of fully correlated risks.
Sorry for the late reply. I don't know how to keep track of responses, other than to revisit old posts when I can remember to.
Robin, you did not object to my characterization of your math as assuming perfectly correlated risks regardless of population size, so I'll assume that is correct. You also didn't try to argue that the assumption is realistic, or that your conclusions are insensitive to the assumption, even though I offered reasons why I suspect both might be false.
I don't know why you talked about the difference between correlated and uncorrelated risks, or the fact that correlated risks exist and is studied in the world of finance. Did I somehow indicate a failure to understand the difference between correlated and uncorrelated risks, or said that I object to any assumption of correlated risks, as opposed to perfectly correlated risks regardless of population size?
Wei, the literature distinguishes correlated and uncorrelated risks, and everyone agrees that near risk-neutrality is appropriate for uncorrelated risks. In the world of finance there are correlated risks, and this literature has been interested in selection of investment strategies in finance. So it is natural and appropriate to focus on modeling the selection of correlated risks.
Will, thanks for the clarification. I think I understand your point now.
Robin, have you had a chance to consider my comments on your and Sinn's math? Whether evolution tends to produce linear-value creatures or log-value creatures (under assumptions that approximate our own past) seems like an interesting and solvable problem, and I hope we can reach some resolution instead of just letting the discussion trail off.
Wei, my point was not to say that we are log utility maximisers because of sex, but that parasitism is a risk that increases in proportion (or some non-linear function) with your lineage (assuming it is dense and connected). Diseases and parasites spread more quickly in a dense than sparse, monocultures bad etc.
Look at sex from the genes point of view not the individuals. It is halving its lineage by adopting sexuality rather than asexuality. A genome having two offspring could have two of 100% its own kids, rather than effectively fostering a random strangers kid. If that is an example of the strategies that an evolutionary lineage maximizing AI adopts, then population of AIs may decide to keep humans around because we are not affected by the same parasitic memes.
We can effectively tell nothing about what an AI trying maximize it's lineage (log or otherwise) will do because we don't know what troubles it will face, because we don't know how it will work.
Robin, I was trying to understand Sinn's math, and didn't noticed that you've introduced novel math of your own. Looking at it now, I can't tell how you derived the conclusion "a wide range of U(x), u(x) goes to log(x) for N large" so I don't know if that is correct or not(*). But from what I can tell, you make the same unrealistic assumption that Sinn did, namely that risks are perfectly correlated within a population no matter how large it is (whereas common sense says that the larger the population, the less correlated the risks its members face), and populations can't split into independent subpopulations. The key phrase here is "if probabilities and choices are independent across timesteps". What happens in your model if whenever a population reaches some threshold size, it splits into two subpopulations each of which then faces an independent set of choices?
(*) Suppose U(x)=x, then doubling r_0 doubles U(prod r_t) but only increases sum u(r_t) by some amount independent of r_1, r_2, ..., etc. So how could U(prod r_t)=sum u(r_t)? You're also taking the expectation value somewhere right, and just omitting it from your notation? There's too many missing pieces for me to figure out what you're doing...
Did anyone look at my math in the post? I think it shows pretty robustly that most any utility over ultimate gains translates to near log utility over each short period relative returns, when there are many not highly correlated periods. The issue isn't whether long term utility is log or linear, the issue is that as long as the future will be long, you are forced to have log utility over short term gains.
Will, sex/parasites is not a good test, because at high population densities parasites are inevitable, and therefore even risk-neutral linear-value creatures will adopt costly defenses like sex (plus I think sexual reproduction has other benefits like allowing a bigger genome to be maintained against mutations so they might adopt sex even without parasites). In other words, sexual reproduction increases expected descendants, not just expected log descendants.
And also, the only way we could have evolved a general cognitive adaption of risk aversion with respect to the number of descendants (which is what Robin cited Sinn's paper for) as opposed to specific, one-off mechanisms like sex, is if there were many different kinds of correlated risk that couldn't be hedged against by maximizing expected descendants and naturally breaking them off into uncorrelated or less correlated subpopulations, but could be hedged some other way which involves a smaller expected number of descendants. That just doesn't seem plausible, given that it's hard to think of even one example.
I'd be interested in continuing the discussion, elsewhere. Although we don't have anywhere else at the moment, here will do for now.
One possible risk associated with being too numerous is the risk from parasites. It is the most convincing evolutionary reason I have heard for sexuality, which halves the lineage of the reproducers genes. Some background.
Whether such risk will exist in the future is another matter.
Having had time to think about Sinn's paper some more, it now seems that Sinn's risk model is less relevant than I first realized. Sinn's prediction (in the sense of statement 1 in my first comment) of risk aversion with regard to the number of descendants only applies to risk that remains perfectly correlated regardless of the size of one's lineage, but one benefit of having more descendants is in fact greater diversification against almost all kinds of risk: you can send your descendants to different corners of the earth or instruct them to exploit different local niches. The only kind of risk in our environment of evolutionary adaptation (EEA) I can think of that can't be hedged against by having more descendants is something like a supernova or asteroid strike large enough to cause global extinction, for which being risk averse wouldn't have made any difference anyway.
As I tried to think of ways to test Sinn's theory, I kept noticing that any candidate test I came up with couldn't either confirm or disprove his theory no matter what the data says because it didn't satisfy his risk model. It seems that Sinn never considered how his theory might be tested, or I think he would have realized how unrealistic his risk model is.
Robin wrote: So they can be terrified of global catastrophe, mildly concerned about car accidents, and completely indifferent to within-lineage tournaments.
No, if we change Sinn's model even slightly, then his prediction breaks down completely. Let's say that risks within populations are perfectly correlated, but when a population gets to a certain size, it splits into two subpopulations such that risks are perfectly correlated within each subpopulation, but independent between subpopulations. This describes car accidents. It's obvious that such groups of populations are essentially equivalent to groups of individual reproducing organisms with uncorrelated risk, so there is no reason to be risk averse.
Hal, unfortunately the specific suggestion you made is too good to be true, but it might be worth think about what other predictions would differ between the two sampling methods, and see if any of them can be tested. That's not quite on topic so if you have any ideas, we should probably discuss them elsewhere.
linear-value creature who dominate in a few rare worlds.
Robin, rare enough to to be mangled in theory?
I meant zero-sum with respect to the weighting on how you sample.Maybe I shouldn't have used that term.
A similar scenario that doesn't select for expectation maximizers is the scenario in which they don't exist.
Hal, I don't think we actually are log maximizers. We are clearly risk averse with respect to physical resources or wealth in general, but that's because there's diminishing return in converting wealth into descendants. (Note that males are much less risk averse than females because males are subject to less diminishing return.) I don't think we are risk averse with respect to the number of descendants, but I don't know where to find the evidence to show that conclusively. One complication is that evolution didn't program us to deliberately maximize either expected descendants or expected log descendants, so we have to determine whether in our environment of evolutionary adaptation we would have had the effect of maximizing expected descendants or expected log descendants. This seems non-trivial but maybe someone can think of a clever way?
Your observation that whether we are log maximizers or linear maximizers might tell us something about the nature of MWI and anthropic selection seems like a really good one. Almost too good to be true, so I'll have to think it over a bit...
Wei, going back to your 1st comment, does the fact that we are log maximizers (supposing that we are, at least we seem to be risk averse while I gather that linear maximizers would be risk neutral) then tell us something about the nature of the MWI and how anthropic selection works? Wouldn't we expect to find ourselves as linear maximizers, according to some models of the anthropic principle?
Robin, if we allow any kind of interaction between the choices of the two lineages, then it becomes a two-player game, and I don't know if we can still prove anything general like Sinn did. However it seems safe to say that the linear-value creatures will still achieve a higher expected population in the long run, and the log-value creatures will still achieve a higher expected log population in the long run. And statements 1 and 2 in my comment above still follow.
Here's a simple example to get a sense of what might happen. Suppose the universe may be type A, in which case it supports at most 10^100 creatures, or type B, in which case it supports up to 10^200 creatures. At time 0 everyone believes the universe is type A with probability 0.9999 and type B with probability 0.0001. Our two types of creatures fight a war at time 0, and they have two strategies to select from. If both sides choose the same strategy, they reach a stalemate, otherwise whoever chooses strategy A wins if the universe turns out to be type A, and whoever chooses strategy B wins if the universe turns out to be type B. It should be obvious that the outcome will be that the linear-value creatures choose strategy B, and the log-value creatures choose strategy A, and they each get what they want, respectively higher expected population and higher expected log population.
Douglas, I don't see how you can assume zero-sum in this case. We have one type of creature that values expected population, and another that values expected log population. Unless you have population1 + log(population2) = constant, which makes no sense as a realistic constraint, you can't have a zero-sum game.
Carl, I'm guessing it's the MWI implications that made Robin rethink.