Augustine’s Paradox of optimal repentance

Eliezer once wrote this about Newcomb’s problem:

Nonetheless, I would like to present some of my motivations on Newcomb’s Problem – the reasons I felt impelled to seek a new theory – because they illustrate my source-attitudes toward rationality. Even if I can’t present the theory that these motivations motivate…

First, foremost, fundamentally, above all else:

Rational agents should WIN.

As I just commented on another thread, this is faith in rationality, which is an oxymoron.

It isn’t obvious whether there is a rational winning approach to Newcomb’s problem. But here’s a similar, simpler problem that billions of people have believed was real, which I’ll call Augustine’s Paradox (“Lord, make me chaste – but not yet!”)

Most kinds of Christianity teach that your eternal fate depends on your state in the last moment of your life. If you live a nearly-flawless Christian life, but you sin ten minutes before dying and you don’t repent (Protestantism), or you commit a mortal sin and the priest has already left (Catholicism), you go to Hell. If you’re sinful all your life but repent in your final minute, you go to Heaven.

The optimal self-interested strategy is to act selfishly all your life, and then repent at the final moment. But if you repent as part of a plan, it won’t work; you’ll go to Hell anyway. The optimal strategy is to be selfish all your life, without intending to repent, and then repent in your final moments and truly mean it.

I don’t think there’s any rational winning strategy here. Yet the purely emotional strategy of fear plus an irrationally large devaluation of the future wins.

I’m not entirely happy with this, because the problem assumes that God cares not just about what you do, but also why you do it, and sends you to Hell if you adopt a strategy for the purpose of winning.

We could say that any general strategy that doesn’t apply only to the paradox itself is admissible. For instance, assume God allows you to adopt a self-identity function that discounts your identification with your future self at such a rate that repentance becomes rational only a few hours before your death, even if Augustine’s paradox was part of your reason. The problem with that strategy is that your life overall would probably not be very winning.

But since the paradox as originally described, including God’s caring about your motives, is a problem that billions of people have believed, we can’t write it off and say “That’s not a fair problem”. As Eliezer has said, Nature doesn’t care if problems are fair. If rationality is always the winning strategy, we must allow all possible Natures; and God is possible.

(Incidentally, googling turns up Augustine’s paradox of time, Augustine’s paradox of teaching, Augustine’s paradox of humility, Augustine’s paradox of memory and learning, and Augustine’s paradox of creation.)

GD Star Rating
loading...
Tagged as: , ,
Trackback URL:
  • http://profile.typekey.com/sentience/ Eliezer Yudkowsky

    My decision theory (not shown) handles situations in which other agents can look at you and extrapolate the decisions that you’ll make in the future, such as Newcomb’s Problem, Parfit’s Hitchhiker and of course the Prisoner’s Dilemma. It explicitly doesn’t handle situations in which other agents read your source code and do things to you that have fully arbitrary dependencies on your source code apart from the decisions thus output.

    We could try to consider fully arbitrary second-order rationality as the algorithm adopted by a first-order maximizing agent under these conditions.

    But then we can always have the God who only rewards you for starting with an algorithm that outputs the first decision in alphabetical order, and who will not reward you for self-modifying to an alphabetic algorithm unless you do so because “alphabetizing” was the first algorithm in alphabetical order that you considered, and who will not reward you if you self-modify “in order to maximize utility”.

    So I explicitly restrict my decision theory to the realms of “other people care about the decisions they predict my algorithm will make under what conditions, but only care about these decisions and don’t care what algorithm makes it apart from that” because this is the problem that turns out to have an elegant first-order solution that doesn’t require precommitment or further self-modification. I’m not aware of any elegant first-order solution to a decision problem more general than this, and so any more general decision problems would have to be handled by an application of the first-order theory to maximize on a second-order choice between algorithms.

    • dmytryl

      Well, suppose I make a copy of you, and give it $1000 , and if it declines, I give real you $10000 . Which I don’t tell to the copy of you. Now some sort of agent that declines $1000, but takes $10000 (being overwhelmed by greed for example) wins, and nobody needs to look at anyone’s source code in any detail, everything can be black boxed.

  • http://elder-gods.org/~larry Larry D’Anna

    I wouldn’t call it faith in rationality; it’s more like a definition. “Rationality” just means the habits of thought that result in win. A peculiar consequence of this definition is that the meaning of rationality depends on the universe outside your mind. In particular if there are gods or aliens that can read and understand your internal state, then the results can be counterintuitive.

  • http://macroethics.blogspot.com nazgulnarsil

    Only Box B seems win-win to me. either I get a million dollars or I proved the predictor wrong (which should be good enough to get me on talk shows and a book deal).

  • http://profile.typekey.com/1227585392s27146/ Philip Goetz

    So I explicitly restrict my decision theory to the realms of “other people care about the decisions they predict my algorithm will make under what conditions, but only care about these decisions and don’t care what algorithm makes it apart from that”

    (I think that positing that the Predictor is infallible, and yet doesn’t know what algorithm you use, is like asking what general relativity predicts will happen to the Earth if the Sun suddenly vanishes from existence. It can’t happen, so you can’t ask the question. Not all gedankenexperimente are allowed. But this only a subsidiary point.)

    My decision theory (not shown) handles situations in which other agents can look at you and extrapolate the decisions that you’ll make in the future, such as Newcomb’s Problem, Parfit’s Hitchhiker and of course the Prisoner’s Dilemma. It explicitly doesn’t handle situations in which other agents read your source code and do things to you that have fully arbitrary dependencies on your source code apart from the decisions thus output.

    Sure. Hence my disclaimer that my paradox relies on God caring about your motives / source code.

    But I think you have to admit that we could all be sims in the computer of a sadistic God. And then you can no longer believe that rationality is always the best strategy. You can at best find some condition which guarantees that rationality is the best policy.

    (If the answer is that black-box rationality, in which the algorithms used by agents are unknowable to anyone, is such a condition, that would be interesting. It wouldn’t apply to the real world; and I don’t think that Newcomb’s paradox could exist in such a world – the required superintelligence could not exist, as noted above. So I think the fact that you’re interested in Newcomb’s paradox implies you would rather have a more general condition. For instance, the stipulation that any knowledge about an agent’s internals is public knowledge, or is possessed by that agent.)

    I’m still curious how your solution works without requiring precommtment.

  • infotropism

    If you start with flawed premises, anything you build upon it will be shaky. An inconsistent system of thought like religious faith will generate cases of action where there is no right thing to do. You can expect idiocy to propagate to all levels of something that’s been built upon a basis laid with one or more nonsensical assumptions.

  • http://profile.typekey.com/1227585392s27146/ Philip Goetz

    “Rationality” just means the habits of thought that result in win.

    It doesn’t. Rationality must have something to do with how you think. Otherwise you should use the term “winning”. It doesn’t appear to me that rationality is always the best policy for humans. (It would certainly be a more competitive strategy if we had no subconscious component to our thinking.) People with overly-optimistic self-evaluations are happier and have better outcomes. In many cultures, rationality would be certain to make you unhappy and likely to get you killed.

    “Rationalism” might, by analogy to other “isms”, mean the dogmatic belief that rational strategies are always the best strategies. A lot of people seem to believe that’s what it means. If that’s what it means, then I’m not a rationalist.

  • Lightwave

    Philip Goetz: Wouldn’t that depend on what each person means by “winning”. If your only value and goal in life is to “be happier and have better outcomes”, then the rational thing to do might be to “have overly-optimistic self-evaluations” (if there really was nothing that would make them even happier).

  • sternhammer

    No, Mr. Goetz, you are misinformed. I don’t think there is any Christian church, conservative or otherwise, that teaches the doctrine you describe.

    Traditional Catholic doctrine was that salvation depends on the grace of God, mediated through the offices of the church. Final thoughts on sinfulness would increase the time of one’s penance in purgatory. But lately they have been soft-pedaling the purgatory business Evangelical christians believe that one moment of sincere belief in Jesus secures salvation. If you have an honest born again experience, then any subsequent sins do absolutely nothing. Calvinists and their progeny (e.g. Presbyterians) believe that one is predestined for damnation or salvation, so last minute thoughts are likewise insignificant. If there is a church you know of that teaches this, please provide an example and a quotation from their books of doctrine.

    Augustine did write about a related problem, namely that one might game the system of righeousness by planning to sin then confess. Hence his description of praying as a young man with a mistress, “God, grant me chastity, but not yet.” You can find that in the Confessions, I don’t have it to hand, but it is in chapter 3 or 4. But I don’t think he discusses salvation in that regard.

    But there are popular literature examples of what you are talking about. In Neal Stephenson’s Quicksilver the Shaftoe boys use that theory to sell their services expediting hangings to condemned criminals. Is that book where you are getting your Christian theology? It is very good, but not as a guide to theology.

    More generally, on whether rationality wins by definition, you guys are wandering in the dark. Read Thomas Schelling’s Strategy of Conflict, chapter 2. There are MANY strategic situations in which rational agents badly underperform irrational ones. Two irrational people — who have honor or loyalty or a belief in a god who watches them — collectively outperform two irrational people in the Prisoner’s Dilemma.

    Moreover, irrationally vengeful agents outperform rational ones as recipients in the Ultimatum Game. See Guth, Schmitberger and Schwarze, 1982. Or there is a good review in Camerer’s Experimental Game Theory.

    Further, irrational males who feel love will tend to outperform rational ones in securing the reproductive labor of discriminating females.

    Irrationality usually outcompetes rationality. That is why there is so much of it. The reason you have emotions is that they evolved to help you outperform rational agents in game theoretic dilemmas. Try reading Robert Frank’s passions within Reason for a good introduction to game theory, evolution and winning strategies based on irrationality.

    A smart, rational person will choose to be irrational at many times. I think cultivating the beneficial types of irrationality is a much smarter strategy than Eliezer’s irrational love of rationality despite the consequences.

    And that includes cultivating religion. Religious people live longer, report far greater subjective happiness, and have more sex than atheists. They also support charities, volunteer, and give blood at much higher rates. That’s why religion is universal among human societies, because it grants such a massive competitive advantage, both in individual game theoretic situations and in fostering social solidarity.

    Why do you guys think it is everywhere? Don’t you realize that societies compete? If atheism gave a society advantages, there would, you know, be a history of atheist societies actually existing for more than, what, 60 years is the current max? And the soviet union didn’t end so well. Modern Europe is a good example of a pretty successful society that is at least effectively atheist. But it is unable to defend itself, relying on religious americans in the military to defend it, and unable to reproduce itself. Maybe a long-term successful atheist society could exist, but so far it hasn’t. That should give you guys pause, at least if you are, you know, rationally updating your beliefs against the historical data.

  • Cyan

    (I think that positing that the Predictor is infallible, and yet doesn’t know what algorithm you use, is like asking what general relativity predicts will happen to the Earth if the Sun suddenly vanishes from existence. It can’t happen, so you can’t ask the question. Not all gedankenexperimente are allowed. But this only a subsidiary point.)

    I’d say that positing that the Predictor is infallible, and yet doesn’t know what algorithm you use is a lot more like positing an oracle machine than like asking what GR predicts upon the sudden vanishing of the Sun. Or more briefly: this gedankenexperiment is allowed.

    It doesn’t [just means the habits of thought that result in win]. Rationality must have something to do with how you think.

    How about “the optimal use of information to choose a goal-directed action”. Implication: acting rationally will win if it’s at all possible.

  • Patrick (orthonormal)

    I think that positing that the Predictor is infallible, and yet doesn’t know what algorithm you use…

    Eliezer said that the Predictor didn’t care what algorithm you used, not that it didn’t know. That is, the Predictor predicts your algorithm, but it puts money in Box B or not depending only on what you decide. That’s how Newcomb’s Paradox differs from Augustine’s Paradox (and Kavka’s Toxin Puzzle, for another example).

  • http://profile.typekey.com/1227585392s27146/ Philip Goetz

    If you start with flawed premises, anything you build upon it will be shaky. An inconsistent system of thought like religious faith will generate cases of action where there is no right thing to do. You can expect idiocy to propagate to all levels of something that’s been built upon a basis laid with one or more nonsensical assumptions.

    The Predictor of Newcomb’s paradox is less likely than a God. So why is it legitimate to ask about Newcomb’s paradox, but not legitimate to ask about God?

    No, Mr. Goetz, you are misinformed. I don’t think there is any Christian church, conservative or otherwise, that teaches the doctrine you describe.

    The second part, that you go to heaven if you sincerely repent (and meet other conditions that may include confession, baptism, etc.) before dying, is taught by almost every Christian church, conservative, Catholic, Protestant, or otherwise. (The major exception is forms of predestination that teach that you can never know whether you are saved or not.) That is the only part that matters for this discussion.

    (Catholic believe that you can go to purgatory even if you die immediately after confession. Therefore, it wouldn’t make any sense for them to place as much importance on confession as they do if they did not believe that dying unshriven condemned you to Hell. The Catholic teaching that babies who die go to Limbo, which was only retracted 2 years ago, would also make no sense. The Catholic position that suicides go to Hell, which is expertly dodged nowadays if you ask about it but is nonetheless the dominant Catholic belief on the subject, would also make no sense. Furthermore, you can find many explicit descriptions of this belief; the most famous is in Hamlet. Regardless of what the Catholic Church teaches, this is what most Catholics throughout history have believed. But, again, it has no bearing on my post either way. Many protestants believe that if you’re saved once, you remain saved. Doesn’t matter WRT my post.)

    Agree about your observations that rationality doesn’t always win. That was the point of my original post.

    Communist states are not atheistic. Communism resembles Christianity and Islam more than Judaism does.

    Philip Goetz: Wouldn’t that depend on what each person means by “winning”.

    Yes. Definition of “winning” is unrelated to the point I was replying to.

  • http://profile.typekey.com/SoullessAutomaton/ a soulless automaton

    Would someone be so kind as to define “rationality” in the context of this discussion? Many of the comments so far seem to be working from a definition that I don’t think matches the one I would use by default.

    For instance, in what way is it reasonable to discuss “irrational” behaviors outperforming “rational” behaviors? Wouldn’t it be, by definition, not rational to persist in behaviors that are demonstrably suboptimal?

  • http://profile.typekey.com/1227585392s27146/ Philip Goetz

    Only Box B seems win-win to me. either I get a million dollars or I proved the predictor wrong (which should be good enough to get me on talk shows and a book deal).

    I did it again. I matched your response to my set of expected misunderstandings based on surface similarity, and initially dismissed it, before realizing what you’d actually said.

    Clever. 🙂

    Patrick wrote:

    Eliezer said that the Predictor didn’t care what algorithm you used, not that it didn’t know. That is, the Predictor predicts your algorithm, but it puts money in Box B or not depending only on what you decide. That’s how Newcomb’s Paradox differs from Augustine’s Paradox (and Kavka’s Toxin Puzzle, for another example).

    Oops! You’re right. Nonetheless, my main point still stands, which is that it is possible that rationality doesn’t always win.

    frelkins: Sorry for my sloppiness. The only thing that matters for my original post is that it is allowable to sin all your life long, and repent in your final moments. This is believed by all Christian sects that I know of, with the exception I noted on predestination, probably because of the thief on the cross, who repented at the last minute and was saved, which I mentioned above. There is disagreement on what caused you to repent (e.g., God or free will), but that’s irrelevant.

    (I hold to my previous comment that Catholics believe that you are damned if you die without confessing. Maybe the Pope doesn’t believe it; maybe a lot of Catholics don’t believe it today; but over the past 1000 years, the evidence I know of convinces me that most Catholics believed it. I said “babies” who die go to limbo; I meant unborn fetuses. That was the official Catholic doctrine until 2 years ago. Either way, as I’ve already noted, that part has no impact on my arguments.)

  • David Ellis


    Religious people live longer, report far greater subjective happiness, and have more sex than atheists. They also support charities, volunteer, and give blood at much higher rates.

    Several problems here. First, self-reported happiness does not sound like a reliable indicator of actual happiness. And the statistics for some things mentioned vary by the poll. One poll conducted by christians, for example, found that atheist doctors volunteer more than theist ones. And as to more sex. That’s not necessarily an indicator of better living. For example, unmarried people typically have less sex than married ones. If theists marry younger than atheists then this alone will probably put them into the category of having more sex on average. But its hardly necessarily a good thing to get married early.

    And then there are the clearly negative numbers. Believers are more likely than atheists, according to many articles I’ve read, to get pregnant outside of marriage, to have abortions, to be child abusers and molesters and to end up in prison.

    The personal advantages of religion are far from as clear as you’ve characterized them.


    “Rationality” just means the habits of thought that result in win.

    Not as I define the term “rational”. They might result in winning more often than not (depending on the variety of win in question). But clearly being rational can also be a detriment—especially in a society where irrationality is exalted and rewarded—which is, to one degree or another, every society I’m aware of.

  • http://blog.contriving.net Dustin

    Philip Goetz: Wouldn’t that depend on what each person means by “winning”. If your only value and goal in life is to “be happier and have better outcomes”, then the rational thing to do might be to “have overly-optimistic self-evaluations” (if there really was nothing that would make them even happier).

    This.

    Many times the “rational” thing to do isn’t the rational thing to do isn’t the rational thing to do.

    Some research shows that those of us who do lots of research into potential product purchases are less happy with their purchase than those who don’t. Thus, the rationally irrational thing to do is not obsess over doing such research.

  • David Ellis


    The optimal strategy is to be selfish all your life, without intending to repent, and then repent in your final moments and truly mean it.

    When the stakes are either eternal joy or eternal agony, it seems to me that the optimal strategy is to be unselfish all your life.

    Even if there were no afterlife I would think a life strongly motivated by kindness would still be an “optimal strategy”. It seems to me an intrinsically better life in its own right than that of an egoist. At the least, I’m not convinced the contrary is true.

  • David Ellis


    Some research shows that those of us who do lots of research into potential product purchases are less happy with their purchase than those who don’t. Thus, the rationally irrational thing to do is not obsess over doing such research.

    Perhaps we should simply conclude that rationality and happiness do not necessarily overlap as much as some would prefer to think.

    Which is fine with me since, while happiness is intrinsically good, its not, to my mind, the only intrinsic good.

  • sternhammer

    Phil Goetz:
    The doctrine that you go to heaven if you repent is not the full extent of your claim. You also claimed that “All conservative variants of Christianity teach, in one way or another, that your eternal fate depends on your state in the last moment of your life.” False. Try and produce some support for modern churches teaching that. You will fail.

    Frelkins. I agree with almost all. But I think the stuff about Mary being co-redeemer is the view of a small minority (lead by a group of Italian ladies in Bayside, Queens. Rejected by the official church doctrinal office during the late years of the last pope, iirc.

    Soulless (love the handle). Eliezer might have his own definition of rational. I am working with the standard game theoretic version which assumes that rational = 1) self interested (ie no social utility, you care about yourself but not others), 2) value-maximizer (that you always want higher payoffs). The definition gets squishy right away once you try to define payoffs. e.g. What is the difference between a rational being with a payoff on receipts to others as part of his utility function and an irrational altruist? The literature goes on and on about that without a resolution. Then there is a big literature defining ways of being irrational. You would like anything Kahneman and Tversky wrote about that, or Max Bazerman has a good basic summary called, um, Judgment in Managerial Decision Making (business schools are the largest audience for this kind of thing).

    But I think it is pretty standard to say that emotions are not rational, e.g. you get mad at someone and chose conflict even when the payoff to surrender are higher. Or you love your wife and stay faithful to her even when there is someone younger and prettier, cause you would feel guilty if you left. In each case, people choose lower payoffs because of a bias in their cognitions. But you can define their payoff structure so that that isn’t irrational, but then once you take that step there is no way to define anything as irrational. Just people rationally following their strange utility structure.

    But it is I think pretty well accepted that no human being are anything like rational in the strict sense. that’s why game theory is more useful as a normative device to understand system dynamics than a prescriptive one to get guidance on what to do.

  • Carl Shulman

    “(I think that positing that the Predictor is infallible, and yet doesn’t know what algorithm you use, is like asking what general relativity predicts will happen to the Earth if the Sun suddenly vanishes from existence. It can’t happen, so you can’t ask the question. Not all gedankenexperimente are allowed. But this only a subsidiary point.)”
    I don’t understand why that’s an impossible gedanken. The Predictor can just set up a separate system to do the predictive analysis (another superintelligence, perhaps) and only output the expected decision of the player. I can find a mathematical result using a calculator, and react to the result without knowing the underlying reasons why that result is right.

  • sternhammer

    David Ellis,

    Yes, one would think that self-report would make the happiness findings unreliable. But self-report correlates very will with 1) estimates of subject happiness by co-workers and associates, 2) measured serum serotinin levels, and 3) suicide, hospitalization and alcoholism rates. If you want a general introduction to that literature Robert Frank has a very good book on consumption and happiness called, um, Luxury Fever. The first couple chapters are all about the problem of measuring happiness. But self-report matches other measures so closely that it is uncontroversial to use it in the cog psych literature I am familiar with. You could also go to Will WIllkinson’s site and search his archives. He blogs a lot about the happiness literature.

    And even if you don’t believe that, regular church attenders live on average 2-3 years longer. That is public data that can’t be faked.

    Yes, the sex stuff has a confounding variable of marriage. But if religion leads to marriage and marriage to sex, then the association still stands.

    You mention some interesting stuff about inter alia belief and child molesting. I am not familiar with that. I think I would have heard of it. But maybe I need to pay more attention to the literature. Could you please provide some cites to back up your claim? The doctor study you mention would be especially surprising. I would love a cite for that. Paul Bloom is an atheist psychologist at Yale who has done a lot of research on the charity gap (and volunteering, blood donation, etc) between believers and atheists. He has a bunch of episodes on bloggingheads talking about that. If you know of data that disproves his studies you could make quite a name for yourself.

  • Yvain

    “I don’t think there’s any rational winning strategy here. Yet the purely emotional strategy of fear plus an irrationally large devaluation of the future wins.”

    No it doesn’t. There are two issues here: the motive for repentance, and the time of repentance. If God rejects anyone who repents solely to avoid going to Hell, then the rationalist who repents out of rational utility maximization and the irrationalist who repents out of raw fear are both damned.

    If God accepts repentance performed solely to avoid Hell, but refuses to accept it as part of a plan to wait as long as possible, then a rationalist will take this into account and realize the rational decision is not to make a plan to wait as long as possible. Ze can get that result by simple multiplication and comparing expected utilities.

    It’s true that a person who irrationally waits as long as possible and then by sheer luck has a change of heart at the last minute will do better than the rationalist, but there’s a low chance of having the sincere change of heart at the last minute, and the rationalist knows this. It’s like the lottery: rationalists who avoid the lottery will end up worse off than the rare irrationalist who plays and wins the lottery, but should on average expect to end up better off by not playing than by playing. If the rationalist knows that honestly having a sincere change of heart at the last minute is actually very likely, ze may take this into account and decide to wait for it.

    So I don’t see this scenario as providing an example of irrational agents winning at all. Therefore, I propose a better scenario:

    “Consider a universe in which God gives people $50 every time they make an irrational decision. Deliberately making irrational decisions for rational reasons does not count as an irrational decision. Maximize your utility in this universe.”

    In this scenario, irrational people really do succeed more. But I don’t think it proves some potential flaw in rationality. The flaw is considering rationality as a choice, rather than a method of deciding between choices. As if there were two buttons in my brain, one reading “Behave rationally” and the other reading “Behave irrationally”, with a little technician inside trying to figure out which button to press, with an infinite regress of buttons inside the technician’s head. In any world where God penalized you for pressing the first button, then the best choice really is to press the second button.

    But we don’t choose that way. The buttons press themselves. In this scenario, my rationality button self-presses and returns the result “The best choice is to have pressed the irrationality button two seconds ago, but it’s too late now and it was never your choice anyway.” The rationality button has returned the correct result, the winning result, the optimum result, as ideally it always should. Unfortunately, it’s useless under the circumstances.

  • sternhammer

    Yvain,

    I think your example of the button pusher in your head pressing rational or irrational is really great. But I think we do make that choice.

    Examples: When you are mad at how somebody treats you, you can try to calm down or go with it and act nuts. That is the basis of the maximizing strategy in an iterated ultimatum game. You can try to inculcate love for an aging spouse, by going on romantic weekends and whatever, or go looking for someone younger. You can try to foster irrational religious beliefs within yourself. I think it is standard for rabbis to tell people who complain about skepticism, “act like you believe and eventually you will.”

    I suppose you can posit an infinite regress, but I think in my head there is only one level.

    I am not saying it always works. I have tried hard to make myself a sincere believer, with limited success. But I have had a lot of successes with cultivating irrationalities too.

  • http://macroethics.blogspot.com nazgulnarsil

    “Rationality” just means the habits of thought that result in win.

    if your algorithm’s performance is worse than random, inserting a randomly decided variable should improve performance. can this random variable be called rationality?

    even if we average the behaviors of all successful people we don’t necessarily have enough trials to say with any confidence that these behaviors are rational given their environment.

  • http://thomblake.com Thom Blake

    The talk of the optimal strategy in this scenario doesn’t make any sense. Whatever you’re measuring, it’s supposed to be infinite after death and finite during life. So if your strategy gives you 1,000,000 utilons and then gets you into heaven, and your neighbor’s strategy gives him -1,000,000 utilons and then gets him into heaven, then you both tie.

  • http://profile.typekey.com/1227585392s27146/ Philip Goetz

    Carl: I don’t think the Predictor can be infallible without knowing something about your algorithm. Saying that the Predictor will use a proxy Predictor is equivalent to partitioning the Predictor’s mind in two, so I still consider it to be the Predictor knowing your algorithm. But, as Patrick pointed out, that doesn’t matter for the purpose that I originally brought it up; my argument was wrong.

    Sternhammer: “You also claimed that “All conservative variants of Christianity teach, in one way or another, that your eternal fate depends on your state in the last moment of your life.” False. Try and produce some support for modern churches teaching that. You will fail.”

    I’ve been in many churches that taught that. I’ve been in no churches that didn’t teach that. By “modern” churches, you probably mean what I would call “liberal” churches.

    As I’ve already acknowledged twice, there are plenty of churches who teach that once you’re saved, you stay saved. And, as I’ve already pointed out twice, that doesn’t matter WRT my post. Let’s please not discuss theology anymore on this thread.

    David: “When the stakes are either eternal joy or eternal agony, it seems to me that the optimal strategy is to be unselfish all your life.”

    Yes, but I’m trying to conduct a thought-experiment here.

    Yvain wrote: “It’s true that a person who irrationally waits as long as possible and then by sheer luck has a change of heart at the last minute will do better than the rationalist, but there’s a low chance of having the sincere change of heart at the last minute, and the rationalist knows this.”

    The person who fears Hell, and discounts the future at a high rate, has a high chance of doing this. There is no tradition saying that God won’t accept repentance done out of fear. Convincing people to repent out of fear is an ancient and strong Christian tradition. It is puzzling how to distinguish this from deliberate planning; but the distinction exists. It’s the same distinction as between premeditated and 2nd-degree murder. Irrational murder is considered more forgivable than rational murder.

    The rest of what Yvain said, I generally agree with. It highlights some confusion in what I originally wrote, and also drags in free will vs. determinism. Can we say that rationality isn’t always the best policy, but it’s the best policy for choosing a policy? That sounds like what I believe, but I’m not convinced that it makes sense. I don’t think I can resolve that question.

    What I believe I have resolved is the question as to whether rationality demands that a rational player win the Newcomb game, on the basis that rationality always wins. The “Augustine paradox” is close enough to the Newcomb paradox, that I can say, If the Newcomb paradox is a situation in which rationality must win, then the Augustine paradox is a situation in which rationality must win. And it doesn’t. Therefore, it isn’t right to assume that rationality must win in the Newcomb paradox.

  • http://profile.typekey.com/SoullessAutomaton/ a soulless automaton

    When discussing whether rationalists should “win” it’s important to distinguish them from three other types of “winners”:

    – Lucky or well-placed fools. Random variation and different starting conditions imply that lots of irrational agents will win for no forseeable reason. If their one success is large enough, or their starting condition good enough, these agents may appear to remain successful even if the decisions they make are no better than chance.

    – Unintentionally correct fools. Successful agents being more influential, an unconcious evolutionary process can be observed in memetic development. Irrational agents will collect a semi-random assortment of common biases, heuristics, and emotional triggers, and for many agents these will encourage relatively optimal behavior in many situations, even though the agent has no deliberate reason for possessing these heuristics and could not justify their existence. Many traditional social behaviors fall in this category.

    – Intuitively insightful fools. Agents with no conscious attempt at rationality, but possessing an intuitive, meta-level understanding of the behaviors of other agents in their social environment, sufficient to work within the environment to obtain more optimal results.

    Irrational agents who do not belong to one of these groups (typically, those in social groups with a higher prevalence of maladaptive heuristic memes) are essentially never “winners”.

    Note that a “winning” rationalist should, as a first order strategy, seek to emulate the actions (but not the mental processes) of the second type of irrational agent. Higher approximations can deviate from the pseudo-irrational baseline if and only if the rational agent perceives a net gain from the behavior compensating for any social cost resulting from deviating from the behavioral norms.

    A rationalist would do well to identify the third type of irrational agent and observe their behavior, as it contains valuable insights on how to improve on the first order stategy above; but this carries the risk of mistaking the first type for the third and being led astray.

  • Nick Tarleton

    soulless automaton: that deserves to be (expanded into) a top level post.

  • Cyan

    The “Augustine paradox” is close enough to the Newcomb paradox, that I can say, If the Newcomb paradox is a situation in which rationality must win, then the Augustine paradox is a situation in which rationality must win. And it doesn’t. Therefore, it isn’t right to assume that rationality must win in the Newcomb paradox.

    Your “Augustine paradox” is essentially a game in which it is a rule that simply making a plan to win is a losing move. There’s nothing like that rule in the Newcomb paradox.

  • http://profile.typekey.com/1227585392s27146/ Philip Goetz

    Cyan wrote:

    Your “Augustine paradox” is essentially a game in which it is a rule that simply making a plan to win is a losing move. There’s nothing like that rule in the Newcomb paradox.

    True. Still orthogonal to my point. Perhaps this is a better statement: If one claims that rationality must win in the Newcomb paradox, because rationality always wins in reality; one must also claim that rationality wins in the “Augustine paradox”, because the “Augustine paradox” is at least as plausible as the Newcomb paradox. But rationality doesn’t win in the Augustine paradox.

  • billswift

    I have been rereading Schelling’s “Strategy of Conflict” and I still think his claimed benefits for irrational behavior in deterrence situations is not compelling. A better strategy would be a rational imitation of “irrational” (meaning less predictable, and often violent) behavior where it would be of benefit, keeping in mind not just the negative consequences of such behavior in the current situation but also the deterrent effect for future situations. For example, if attacked, retaliate immediately with utmost viciousness, not just for current defense, but more importantly so that no one will attack you in the future (eg, Afghanistan; unfortunately Iraq undercut the deterrent effect of Afghanistan, since it indicated the US might attack you even if you didn’t attack the US first). Also, even Schelling put only relatively limited value on apparent irrationality, even for deterrence.

  • Cyan

    It’s not orthogonal — I should have been less succinct. Here’s the slightly longer version: the flaw in the statement “rationality always wins in reality” is that sometimes winning just isn’t possible; the correct statement adds the proviso “if possible”. Rationality (arguably) doesn’t win the Augustine paradox because its rules say no planning — it directly targets a central feature of rationality. The Newcomb problem does not target rationality, so the outcome of rationality as applied to the Augustine paradox has no relevance to predicting the outcome of rationality as applied to the Newcomb problem.

  • http://socraticgadfly.blogspot.com SocraticGadfly

    When I was in diviinity school, I had an old prof (a tanker in WWII, no less) whose ideas of Muslim conversion were based on this line of reasoning.

    He said we should hold a sword up to each individual’s throat, get them to convert, then chop off their heads.

    Of course, as civil libertarians know, and neocons don’t, as sadly shown at Bagram, Abu Ghraib and Gitmo, a sword, or far worse, just puts words on someone’s lips and not beliefs on their insides.

  • michael vassar

    It seems to me that the basic point here is that as follows.
    a) The Newcomb paradox is a paradox because it is a situation where agents are penalized specifically for following a particular strategy mandated by a particular conception of rationality.
    b) This is used to argue that the conception of rationality that is being penalized is not “really rational” as a “really rational” strategy would involve responding to the fact that one is in a situation where that behavior is penalized and acting accordingly.
    c) But ANY pattern of behavior can be specifically penalized by an outside agent. For instance, a “counterfactual predictor” who put the money in both boxes only if you would two box in Newcomb is not obviously less plausible than a Newcomb predictor.
    d) Yet to win in the above situation you should be the sort of person who two-boxes in Newcomb. No behavioral strategy always wins.

  • http://profile.typekey.com/1227585392s27146/ Philip Goetz

    Mike – I think that the Newcomb paradox assumes you know ahead of time that you’re going to be faced with the Newcomb Predictor, and get to optimize your decision strategy for it. Otherwise it doesn’t present a paradox; you just lose.

    Cyan wrote:

    Here’s the slightly longer version: the flaw in the statement “rationality always wins in reality” is that sometimes winning just isn’t possible; the correct statement adds the proviso “if possible”.

    But the irrational, short-sighted, emotional person regularly wins at the Augustine paradox. So winning is possible.

    The Newcomb problem does not target rationality, so the outcome of rationality as applied to the Augustine paradox has no relevance to predicting the outcome of rationality as applied to the Newcomb problem.

    I’m not trying to predict the outcome of rationality in the Newcomb problem. I’m not claiming that rationality loses in the Newcomb problem. I don’t think I can state what I’m trying to show any clearer than I already have.

  • http://www.acceleratingfuture.com/tom Tom McCabe

    “I’m not entirely happy with this, because the problem assumes that God cares not just about what you do, but also why you do it, and sends you to Hell if you adopt a strategy for the purpose of winning.”

    “It explicitly doesn’t handle situations in which other agents read your source code and do things to you that have fully arbitrary dependencies on your source code apart from the decisions thus output.”

    From my guess about Eliezer’s theory (http://www.acceleratingfuture.com/tom/?p=159), it should be able to handle God rewarding you for being in a state of mind where you will wind up repenting, or even God rewarding you for belief in Cthulhu (assuming Cthulhu doesn’t exist). The problem with this is that it creates an infinite recursion; if you try to use a second-order algorithm to determine which first-order algorithm to use, God will see that you’re making second-order decisions based on utility for yourself and send you to Hell. If you use a third-order algorithm to choose a second-order algorithm, God sees it and sends you to Hell, etc., etc.

  • http://profile.typekey.com/sentience/ Eliezer Yudkowsky

    Tom, your guess is wrong. (I observe that you are usually wrong when you try to guess things that I haven’t said; you should be aware of this.) My theory is first-order only – that’s kinda the point, that it doesn’t need to modify itself because it’s just doing the right thing to begin with, that is, it is reflectively stable. It handles Newcomblike problems where other agents care what you will do and can predict what you will do, but the other agent must behave invariantly with respect to your algorithm once this information is screened off; the theory doesn’t handle agents who care about arbitrary features of your source code, not without further reflection.

  • Cyan

    So winning is possible.

    But not for a rational agent, by construction. This is similar to the Godel-type sentence, “Cyan cannt prove this sentence.” You can prove it easily, but I never can, even though it is a true sentence. (But don’t take my word for it…)

    Suppose the postulate were that you always win at checkers, because you always win every game it is possible for you to win. Must one therefore also claim that you must win in a game with the rule “People with a ‘z’ in their last name lose automatically”? Does the fact that you lose at this game have any bearing at all on the postulate?

    I’ve done my best to preserve the phrasing of your comment at 10:17PM in the hopes of making my point congruent with yours.

  • sternhammer

    Phil,

    I am not referring to liberal churches. I am referring to the catholic and conservative evangelical churches that I have attended as a member (switched after getting married). Their doctrine explicitly disavows what you attribute to them.

    You say that you have been in the churches and that is what they believe. I disagree. I think that is what you thought or heard when you were there, not what they were actually saying. An anecdotal, gestalt sense from an unsympathetic visitor is not the best guide to what doctrine any group actually believes. You are generally right that repentance and forgiveness are big topics. I think you extrapolated to what you thought were the logical conclusions, but they don’t think are.

    If you want to Overcome Bias, we need to test whether one of us was inaccurate in understanding what was heard. That is why I think we should go to citing written doctrine, which they will helpfully provide on their websites. They also produce enormous amounts of books and articles which you could quote to make your point. You are not making that move. I wonder why.

    Billswift: your idea that rational people could get most of the benefits of irrational deterrence sounds great in theory. It breaks down in practice. There were a bunch of interesting studies on lie detection. Most people are very weak in detecting factual lies (something like 52% accuracy when 50-50 is the baseline). But people are awesome at detecting misrepresentation of emotional states (like 90%). Have you ever tried to pretend that you weren’t upset and someone knew that you actually were? That’s your problem. Evolution has been working on this problem for millions of years, and we have developed a lot of hardware and software for handling it.

    That’s why FBI and CIA manuals about lie detection focus on emotional states, not factual errors. They look for your move, because it is easier to spot.

    It turns out the best way to fake irrational states is to actually have them. And people are great at getting themselves into irrational states.

    One example of this is in mating. Females invest much more labor and bodily integrity in reproduction. So they sift the population for males who are irrationally loyal to females and children. So some males try to fake irrational attachment (we call it love). Now maybe you are a great faker, and some guys out there can falsely talk a lot of girls into bed. But the world we observe is one in which detectible irrational attachment is strongly favored, and exists everywhere.

  • Will Pearson

    Eliezer wrote:”My decision theory (not shown) handles situations in which other agents can look at you and extrapolate the decisions that you’ll make in the future, such as Newcomb’s Problem, Parfit’s Hitchhiker and of course the Prisoner’s Dilemma. It explicitly doesn’t handle situations in which other agents read your source code and do things to you that have fully arbitrary dependencies on your source code apart from the decisions thus output.”

    This is not the situation we have been in during evolution. Or at least nature cares what algorithms you use when deciding your survival, it doesn’t have arbitrary dependencies on the the code as such. E.g. an animal that uses quicksort vs insertion sort for its sorting needs will use less energy thinking and have more available for other purposes.

  • http://www.allancrossman.com Allan Crossman

    Phil: your eternal fate depends on your state in the last moment of your life

    Ignoring the uninteresting question of whether Christians actually believe this… the winning strategy is to do whatever you can to increase the probability of you having a winning “state” at the final moment.

    The optimal self-interested strategy is to act selfishly all your life, and then [sincerely] repent at the final moment. (“sincerely” added)

    This is the optimal action. But since winning requires sincerity, it is not the optimal strategy. Planning to be insincere for most of your life is not the best way of achieving the goal of actually being sincere at the end.

    The optimal strategy is to do whatever you can to become sincere. (If that isn’t allowed either, well, you’re screwed – the game has been set up so that anyone in your initial state loses.)

  • billswift

    I’m not sure “winning” is a useful word except in sports (and other zero-sum games).
    The phrase “best available end-state” would be more accurate for real world situations.

  • http://t-a-w.blogspot.com/ Tomasz Wegrzanowski

    Further, irrational males who feel love will tend to outperform rational ones in securing the reproductive labor of discriminating females.

    I disagree with that and I’d say dating is one of the best success cases of rationality – there’s nothing natural (=similar to original ancestral environment) about that, the so called “players” have made a small private science of out it, and they’re just many orders of magnitude better at it than random guys who do it irrationally on genetic intuition.

    If you care about genes and not women – you can cheat even harder and become a sperm donor, it can score you >10 kids easily at tiny fraction of effort.

  • David Ellis


    And even if you don’t believe that, regular church attenders live on average 2-3 years longer. That is public data that can’t be faked.

    Regular church attenders vs Unbelievers?

    Selecting out the believers who aren’t serious about and committed to their principles while not doing the same in regard to nonbelievers seems a shaky methodology.

    You are aware, are you not, for example, that there are atheists who attend churches? Quite a lot of Unitarian Universalists are nontheists.

    So wouldn’t it make more sense to compare atheists who attend church with theists who attend church and see if its not the believing that’s the important factor?


    You mention some interesting stuff about inter alia belief and child molesting. I am not familiar with that. I think I would have heard of it. But maybe I need to pay more attention to the literature. Could you please provide some cites to back up your claim?

    religiosity and racial prejudice:

    http://www.jstor.org/pss/3512331

    Religiosity and sexual crime:

    http://www.springerlink.com/content/6wnv26q344t52hw7/?p=ed5a99b4443a41a5b36114e7acbfc8b3&pi=5

    http://schwitzsplinters.blogspot.com/2007/02/do-atheists-commit-less-vicious-sexual.html

    Abortion rates in regions of high religiosity:

    http://bhascience.blogspot.com/2008/10/religion-and-abortion-facts.html

    That’s just a few things that a couple of minutes of Google searching turns up. I’m certainly no expert on the topic and don’t claim to be. What I am suggesting, though, is that the evidence does not seem to be as clearly in favor of religion being a positive force in society as you have implied. You seem to be cherrypicking the evidence for things that support your preferred position.

    Personally, it doesn’t matter to me much either way. I don’t form my beliefs based on a desire for beliefs that get me more sex or make me live longer—much as I like both of those things. I’m interested in forming my beliefs by methods with the best chance of generating true beliefs and a low chance of generating false ones.

  • David Ellis


    And even if you don’t believe that, regular church attenders live on average 2-3 years longer.

    Another point about being careful with correlations. What if this is not because religiosity makes you live longer but simply reflects something like “those who attend church regularly drink alcohol, on average, less than those who don’t”.

    If so, as a nondrinking atheist, I’m likely to have the benefit already without having to put irrational beliefs into my head to get it.

    We need to show great care that our preferences aren’t influencing the way we judge data like this.

    My main point is not that religion doesn’t have positive benefits in some ways. Maybe it does. In fact, I think it probably does (though I’m far from convinced there aren’t secular alternatives that can achieve equally good results). My point is that I don’t think you’re being very conscientious in regard to how you’re evaluating the evidence. I pointed out some very obvious things you really should have though of yourself but seem to have entirely missed in your haste to find positive social and personal benefits with religiosity

  • Jordan

    “Yet the purely emotional strategy of fear plus an irrationally large devaluation of the future wins.”

    This is the flaw in your argument, I believe. Devaluations are akin to valuations, they are simply part of your overarching utility function and are neither rational nor irrational. Someone who prefers 1 utilicron today to 100 tomorrow isn’t irrational. The problem is “utilicrons today” and “utilicrons tomorrow” aren’t pure utilicron units, they still have to be processed one final time by a utility function to shed the time component. In this case our agent’s utility function may well perform the final computation and conclude “1 utilicron today” = “1 utilicron”, “100 utilicron tomorrow” = “0.5 utilicron”. That agent would be acting perfectly rationally to live in sin its whole life up until the moment it was about to die.

    What your argument has really shown then is that multiple rational agents with different utility functions will ultimately arrive at different net utility, and indeed an agent may perform better than another agent even based on that other agent’s utility function. That isn’t really surprising considering the setup of this problem essentially becomes “If you have Utility Function 1, you lose. If you have Utility Function 2, you win.” In my opinion this has nothing to do with being rational or not.

  • http://www.walljm.com/ Jason Wall

    I don’t know if anyone has mentioned it yet, but the example used in the original article fails to take into consideration the full christian doctrines. You assume that salvation is the end all. Simply avoiding hell isn’t the end goal. Salvation has two other largely significant purposes:

    1) The relationship provided (in the Christian context) with Christ enriches existence. Salvation grants the supplicant sonship, access to God via the mediator Jesus Christ, a host of promises, and the gift of the Holy Spirit.

    2) There are rewards promised to the Christian that have nothing to do with simply avoiding hell. A Christian’s rank in the world created after Jesus returns, the wealth and richness of his life on the new earth is determinant by how much he suffers for Christ’s sake.

    The bible teaches that the pleasure derived from a “sinful” action is short lived, that ultimately, the better life is lived when a person avoids such short term enjoyments in lui of richer gains later.

    Thus, a rational Christian will deny himself and seek to do good works, not because his salvation depends on it, but because doing such works earns him an eternal reward, such rewards being of far greater value than any temporal gains derived from sinful behavior.

  • Ben Jones

    Perhaps this is a better statement: If one claims that rationality must win in the Newcomb paradox, because rationality always wins in reality; one must also claim that rationality wins in the “Augustine paradox”, because the “Augustine paradox” is at least as plausible as the Newcomb paradox. But rationality doesn’t win in the Augustine paradox.

    Well, yes, congratulations. Now substitute ‘the Augustine Paradox’ for ‘game contrived such that rationality loses’ and reread the last sentence in the paragraph above….

    This reminds me of one of Eliezer’s posts on the use of random noise in AI. If the environment is not just unknown but actually out to screw you up, strange things happen. If the environment includes an omnipotent entity who targets rational thought, well, hell, think what you want. It’s not difficult to think up a scenario in which rationality is punished. You just say ‘in my game, you get punished for rational thought.’ But it’s not that useful, and certainly not revelatory. If you’re hoping to prove that there is no universal strategy, then congratulations, trivial job done. But if you really want to impress me, show me a real-world, god-free situation in which (reciprocal) rationality loses. (And no, ‘being happy’ doesn’t cut it.)

  • D Bachmann

    sternhammer’s comment above (March 1, 06:14 PM) is excellent, and would probably deserve to be elevated to a topic in its own right.

    Rationalists should view irrational allegiences such as religion with admiration, not with contempt. To a rationalist, religion should exhibit a similar beauty as do other evolved organs like, say, the inner ear or the liver. Now, it is possible to admire your inner ear as a marvel of evolution while still using it, but it is more difficult to admire religion as the sleek, mean, optimized tool it is and still be religious. That’s because religion and rational thought both impose on your cognition, and you will tend to run into mental resonance catastrophes if you try to be religious and be rational about being religious at the same time. Hence the big divide between rationalists and people of faith, but there are, nevertheless, exceptional people who manage to reconcile the two.

    But let us not forget that rationalism itself has huge advantages in the competition between societies. The problem is that these may be stolen by irrational groups. Thus, the nuclear bomb isn’t possible without a stable tradition of rationalism, and once you have the bomb, you can safely scoff at the irrational theocracies or warrior societies with all their valour, self-sacrifice and loyalty. Until they steal the design from you, that is.

  • http://shagbark.livejournal.com Phil Goetz

    Jason wrote:

    Devaluations are akin to valuations, they are simply part of your overarching utility function and are neither rational nor irrational. Someone who prefers 1 utilicron today to 100 tomorrow isn’t irrational.

    Correct. I was speaking in shorthand. If you believe that rationality always wins, this should not depend on your valuations. Hence, even if you can win at Augustinian repenting by choosing a particular set of values, it doesn’t suffice.

    sternhammer wrote

    I think that is what you thought or heard when you were there, not what they were actually saying. An anecdotal, gestalt sense from an unsympathetic visitor is not the best guide to what doctrine any group actually believes.

    I was a fundamentalist evangelical Christian for about 15 years. I know what I’m talking about, in great depth.

    Jason Wall wrote:

    Simply avoiding hell isn’t the end goal.

    Yes. But I’m conducting a thought experiment. I don’t mean it as an attack on Christianity. Tho I can see how it could be seen that way.

    Ben Jones wrote:

    But if you really want to impress me, show me a real-world, god-free situation in which (reciprocal) rationality loses. (And no, ‘being happy’ doesn’t cut it.)

    As I’ve posted repeatedly above, I am trying to show that, IF YOU BELIEVE RATIONALITY SHOULD WIN THE NEWCOMB PROBLEM because rationality always wins in real life, THEN you should also believe that it will win in the Augustinian paradox, which, unlike the Newcomb problem, is believed by many people to be a real-life problem. If you’re asking for a real-world, god-free situation (which may be the right thing to do), then you dismiss both paradoxes.

    Cyan wrote:

    Suppose the postulate were that you always win at checkers, because you always win every game it is possible for you to win. Must one therefore also claim that you must win in a game with the rule “People with a ‘z’ in their last name lose automatically”? Does the fact that you lose at this game have any bearing at all on the postulate?

    A more precise analogy would be: Suppose E says that a rational agent can win at the Newcomb game, because a rational agent wins all games that might take place in the real-world that it is possible to win. I then provide a different game, and show that it is much more likely to be real-world game than the Newcomb game, and show that a non-rational strategy can win at this game, yet a rational strategy doesn’t. I’m playing a bit of a trick, because the game is formulated to penalize rationality; but that doesn’t violate the conditions E originally laid down, and doesn’t step outside the boundary of “real world games that it is possible to win”.

    This doesn’t show that a rational agent can’t win at Newcomb. It doesn’t show that either game will actually take place in the real world. It doesn’t show that rationality is good or bad, or always wins, or doesn’t always win. It JUST SHOWS that THE A PRIORI BELIEF THAT RATIONALITY MUST SOLVE THE NEWCOMB PARADOX IS UNJUSTIFIED.

    Has anybody understood that this is what I’m trying to say? Anybody?

  • http://shagbark.livejournal.com Phil Goetz

    Tom, your guess is wrong. (I observe that you are usually wrong when you try to guess things that I haven’t said; you should be aware of this.)

    I have a wonderful proof that a^n + b^n = c^n has no integral solutions when n > 2. Unfortunately I am too busy to write it down. Until I do, please don’t try to guess what it might be.

  • http://shagbark.livejournal.com Phil Goetz

    Only Box B seems win-win to me. either I get a million dollars or I proved the predictor wrong (which should be good enough to get me on talk shows and a book deal).

    I did it again. I matched your response to my set of expected misunderstandings based on surface similarity, and initially dismissed it, before realizing what you’d actually said.

    Clever. 🙂

    Oops. I thought you meant “only box A”. Which wouldn’t get you a million dollars. And isn’t allowed by the game.

  • billswift

    Irrational and emotional are NOT the same. Although I must admit, many people, esp religious, conflate them.

  • Nick Tarleton

    As I’ve posted repeatedly above, I am trying to show that, IF YOU BELIEVE RATIONALITY SHOULD WIN THE NEWCOMB PROBLEM because rationality always wins in real life

    Is this even what “Rational agents should WIN” means? I read it as a normative statement, roughly “don’t worry about your ritual of cognition except insofar as it works or doesn’t”, not an empirical prediction.

    the “Augustine paradox” is at least as plausible as the Newcomb paradox

    The literal version of either is very unlikely, but the Newcomb paradox resembles the (common) single-shot prisoners’ dilemma; is the Augustine paradox similarly analogous to anything?

  • http://shagbark.livejournal.com Phil Goetz

    The literal version of either is very unlikely, but the Newcomb paradox resembles the (common) single-shot prisoners’ dilemma; is the Augustine paradox similarly analogous to anything?

    I don’t think so.

    But the single-shot PD isn’t regarded as a paradox. No one says “A rational agent should get the maximum payoff in the PD.” So I don’t think it matters.

    Clever. 🙂

    Oops. I thought you meant “only box A”. Which wouldn’t get you a million dollars. And isn’t allowed by the game.

    Oops. What you actually said was more clever than that. I misinterpreted the same simple comment twice. 😛

  • sternhammer

    Phil, I am sorry that you are frustrated that no one is responding to your main point. I found a bunch of issues you raised more interesting than the one you are concerned with. I am happy to endorse your view that an a priori belief that rationality will solve the Newcomb Paradox is unjustified.

    I want to continue on your claims about christian theology. Some people don’t care about this. Coll. I have two reasons for caring.

    1. I think the mental hardware that supports animosity between religious groups is still present in people who don’t espouse a religion. Becoming an atheist doesn’t rid you of the stuff that makes irish catholics and protestants hate each other. I think a lot of people at OB display the same sort of hostility that we see in community conflicts, as part of the Atheist Tribe against the Believer Tribe. One way that manifests itself is in inaccurately ascribing silly beliefs to the Other. I believe you that you are not intending to attack christianity. I do think you are falsely claiming they have a belief they don’t. I think showing that you are making that error is valuable to update this community’s beliefs about the infallibility of their own beliefs about other groups.

    2. We have a perfect example of differing empirical beliefs. You say “evangelicals think X.” Frelkins and I say, “no they think Not X.” We are each making implicit arguments about how to go about the task of finding which claim is accurate. I think it is an interesting exercise to try to reach agreemnt on that.

    You started by saying, “All conservative variants of Christianity teach, in one way or another, that your eternal fate depends on your state in the last moment of your life.”

    I said, no they don’t, and gave what I think the actual beliefs of various conservative christian subgroups. You were not convinced.

    Frelkins also said your claim was innacurate, and he made the better move of quoting the bible verses that evangelicals rely on in their arguments with catholics to support a view inconsistent with your claim. You were still not convinced.

    I asked you to provide some supporting quotation or citiation that supports your view. You didn’t. As Eliezer’s Super Happies would say, “If you do not give us that information, we will take into account the fact that you do not with us to know it.” If you were able to successfully do that, that would resolve at least one part of our dispute, namely that there are at least some conservative christians, not all, that take your view. You can still make that move at any time if you are able.

    Instead you simply stated that “I was a fundamentalist evangelical Christian for about 15 years. I know what I’m talking about, in great depth.” The reason this is unconvincing is that many other people have personal experience of those churches, and came to a directly opposite conclusion than you. Frelkins and me are two of them. More generally, I think appeals to personal experience tend to be unconvincing because they are not testable or replicable. Though I am happy to accept that you know many things about evangelical christians accurately, I think this is not one of them.

    So, how to reach agreement. I think when we have an empirical disagreement the thing to do is to jointly design a test that will resolve it. When the empirical disagreement about the subjective beliefs of others, asking them is probably the best we can do.

    Here’s what I propose: you provide the name and location of the church you attended for 15 years. That should only take you 10 seconds. I will contact the pastor and ask him to post a comment here, with enough identifying information to convince people that it is actually him.

    The question is: Phil Goetz has claimed that “All conservative variants of Christianity teach, in one way or another, that your eternal fate depends on your state in the last moment of your life.” Is that in fact the doctrine of your church?

    I will post the whole of my email to him on the site, so people can check it for biasing influence on the subject.

    I know Eliezer loves prediction markets. I am willing to bet you $100 that your former pastor says you are wrong. And we can make this a prediction market by taking side bets from others. I can cover anyone who wants to bet on Phil up to $10 per person, to a max of say 20 side bets. Maybe you could cover action from anyone who wants to bet on my view matching your cleric’s. You can name different amounts if you want, and I will match yours if they are lower or not much higher.

    So, the questions to you are:
    Do you agree to my proposed test?
    Do you propose a different test?
    Are you willing to bet me?
    Do others want to bet one way or the other?

    David Ellis has some good posts that raise complex questions of moderating and mediating variables skewing results. I’ll try to make a good rsponse to that later.

    D Bachman, thanks. I love the questions you raise. I think there are great advantages to both rationality, and well-chosen irrationalities. But I don’t think we have to choose between them, as the history of your a-bomb example shows. The US is both the most rationally capable, technologically advanced society in the world, and one of the most religious (I think we are #2 after India; but easily the most religious in the developed world). I don’t think they are incompatible in the same societies or in the same individuals. The most brilliant scientist I know is also a total raving fundie. He firmly believes that god created the world in 4000BC, but he is a highly successful research oncologist. He clearly knows all about cell evolution, because he is so good at manipulating it, but that just goes into a different category in his brain. I think the big divide you notice between rationalists and people of faith is more tribal than cognitively necessary.

    But maybe there is a problem thinking rationally about religion and also successfully believing. At least it is hard for me.

  • sternhammer

    Billswift,
    “irrational and emotional are not the same.” Can you explain what you think the differences are?

  • http://www.walljm.com/ Jason Wall

    Phil Goetz: “Yes. But I’m conducting a thought experiment. I don’t mean it as an attack on Christianity. Tho I can see how it could be seen that way. “

    i didn’t read your post as an attack on Christianity (though i did pickup that you don’t consider yourself Christian any longer). My point was that, the thought experiment, the paradox, as you phrased it was, “The optimal self-interested strategy is to act selfishly all your life, and then repent at the final moment” and that is wrong.

    The optimal self-interested strategy isn’t to act selfishly all your life, unless the end goal is simply avoiding hell. If your end goal is maximizing enjoyment in life, then living a righteous life is the optimal self-interested strategy.

    the rest of your argument relied on that statement. and it is false, rationally speaking.

    Sternhammer:

    I get what you’re trying to say, but i believe you misread Goetz. When Goetz saed, “All conservative variants of Christianity teach, in one way or another, that your eternal fate depends on your state in the last moment of your life”, thats true. his followup statements, “If you live a nearly-flawless Christian life, but have a sinful thought ten minutes before dying and the priest has already left, you go to Hell. If you are sinful all your life but repent in your final minute, you go to Heaven”, should be read as an either/or. Some evangelicals believe you can loose your salvation (typically you find this in the more charismatic denominations), just as he described in his first statement.

    Either way, your fate does depend on your “state” in the last moment of your life. even if that “state” was decided 80 years ago or 5 minutes ago. Its true for the “once saved, always saved” crowd.

  • http://profile.typepad.com/walljm Jason Wall

    (i posted this… and it showed up. but the comments were different. then i signed in under my typekey username and my comment went away. so i’m reposting it. sorry if it got lost, or posted twice.)

    Phil Goetz: “Yes. But I’m conducting a thought experiment. I don’t mean it as an attack on Christianity. Tho I can see how it could be seen that way. “

    i didn’t read your post as an attack on Christianity (though i did pickup that you don’t consider yourself Christian any longer). My point was that, the thought experiment, the paradox, as you phrased it was, “The optimal self-interested strategy is to act selfishly all your life, and then repent at the final moment” and that is wrong.

    The optimal self-interested strategy isn’t to act selfishly all your life, unless the end goal is simply avoiding hell. If your end goal is maximizing enjoyment in life, then living a righteous life is the optimal self-interested strategy.

    the rest of your argument relied on that statement. and it is false, rationally speaking.

    Sternhammer:

    I get what you’re trying to say, but i believe you misread Goetz. When Goetz saed, “All conservative variants of Christianity teach, in one way or another, that your eternal fate depends on your state in the last moment of your life”, thats true. his followup statements, “If you live a nearly-flawless Christian life, but have a sinful thought ten minutes before dying and the priest has already left, you go to Hell. If you are sinful all your life but repent in your final minute, you go to Heaven”, should be read as an either/or. Some evangelicals believe you can loose your salvation (typically you find this in the more charismatic denominations), just as he described in his first statement.

    Either way, your fate does depend on your “state” in the last moment of your life. even if that “state” was decided 80 years ago or 5 minutes ago. Its true for the “once saved, always saved” crowd.

  • http://shagbark.livejournal.com Phil Goetz

    sternhammer,

    No, I don’t want you bothering my ex-pastors. And I know perfectly well that you can get different answers from them depending on how you ask the question. I will not bet money on their rationality and logical consistence.

    I already wrote, “The second part, that you go to heaven if you sincerely repent (and meet other conditions that may include confession, baptism, etc.) before dying, is taught by almost every Christian church, conservative, Catholic, Protestant, or otherwise. (The major exception is forms of predestination that teach that you can never know whether you are saved or not.) That is the only part that matters for this discussion.”

    I also explained some of the reasons behind my opinions on particular Catholic beliefs. And I referred, twice, to the episode of the thief on the cross as the explanation of why all Christians believe you can be saved at the end of your life regardless of how you have lived your life before that.

    You seem unaware that what you are claiming, in light of what I have said, is that most Christian churches teach that you can still go to Hell even after you have sincerely repented. You keep repeating my first statement and ignoring all of my later clarifications and qualifications. You keep asking for responses I have already made. I can only conclude you have written a long impassioned response to me without reading what I said.

  • sternhammer

    Phil,

    I disagreed with one particular sentence you wrote, quoting it several times. At no point were you able to admit that you were wrong about that. I think you are hinting that you now know it is wrong. But I am not sure. Your post seems very oblique.

    Perhaps you can clarify. Do you, now, believe as you earlier wrote, that:

    “All conservative variants of Christianity teach, in one way or another, that your eternal fate depends on your state in the last moment of your life. If you live a nearly-flawless Christian life, but have a sinful thought ten minutes before dying and the priest has already left, you go to Hell.”

    Try to answer yes or no.

    If no, then we have no argument.

    I have read your last paragraph many times without understanding what you are saying. Please edit it for clarity. Are you talking about something I have claimed or something you have claimed?

    And when you reject my proposed test as biased, despite my offering you the exact language I would use and your alleging no bias in that, perhaps you would consider offering a test of your own.

    But not offering a test, and not offering supporting evidence that can be checked, would be very consistent with an effort to avoid detection of errors in your claims. I can’t read your mind. But I spent years as a lawyer, and when a witness acts like that, it speaks louder than anything he says.

  • Douglas Knight

    Nick Tarleton,
    how is PD like Newcomb?
    Newcomb seems much easier to me than PD.

    Phil Goetz,
    Lots of people responded before your hysterical outburst. I would have, too, if sternhammer hadn’t hijacked the thread. Eliezer, in the very first comment, lays out his position quite clearly. In particular, he hints that he does believe that “A rational agent should get the maximum payoff in the PD.” The absolute max is impossible from symmetry, but he has said elsewhere on OB that two transparent rational agents cooperate in a one-shot PD.

    You seem to be upset that he didn’t address the meaning of the slogan “rational agents should win.” This is stupid to argue about. I endorse Nick Tarleton’s version.

  • Douglas Knight

    I disagreed with one particular sentence you wrote, quoting it several times. At no point were you able to admit that you were wrong about that.

    sternhammer:Phil::Phil:Eliezer

  • asdf

    @soulless:

    – Intuitively insightful fools. Agents with no conscious attempt at rationality, but possessing an intuitive, meta-level understanding of the behaviors of other agents in their social environment, sufficient to work within the environment to obtain more optimal results.

    Do you meant to suggest here that rationality requires consciousness or that an intuitive process that was provably optimal in all circumstances would not be a rational process? It sounds like it. If so, this seems problematic, as I hope the following thought experiment shows.

    Consider two people whose brains are working on a complex problem (that doesn’t require and isn’t aided by consciousness of any form), with the only relevant difference between what the 2 brains are doing being that one has conscious awareness of (some of) what it is doing as it works on the problem. (If you think this is impossible in principle, please say why.) If they both reliably solve the problem optimally — there is no other strategy of any kind that is superior to either one — via an analogous sequence of steps and calculations, on what basis can you say that one is rational and the other is not?

    I don’t think intuition ever gets this reliable, and so it is not rational for that reason, but if it could be so reliable (which seems possible in principle for domains that don’t involve consciousness [e.g., anything a computer could do optimally], although even then, perhaps consciousness could be modeled ‘intuitively’ in a non-conscious manner), then it would be a form of rationality.

    What would you call a state in which you have internalized the principles and rules for a domain of problems that you can now solve instantly without any conscious effort whatsoever? It sounds like a form of intuition to me. And yet, it’s hard to say that the process was a form of rationality for the entire time that it required effort and was conscious, but that after you became so skilled at the intellectual tasks that they just began to happen automatically and effortlessly, the process suddenly became non-rational.

  • asdf

    The details of planning, sinning, and Christianity are all irrelevant to the underlying point. I think the following is a simpler version of Phil’s scenario that illustrates the point.

    Let us suppose that God decides who goes to heaven and who goes to hell using the following rule: you go to heaven if and only if you have never been rational with respect to heaven or hell, otherwise you go to hell.

    Obviously, a rational agent always loses this game, and non-rational agents always win.

    I think that what scenarios like this show is that you can’t say just “rationality always wins”, or more carefully, “no strategy has a better expected outcome than rationality”. Instead, you have to say that “no rationally chosen strategy has a better expected outcome than rationality”, or “no strategy has a better expected outcome than rationality unless the game includes an a priori penalty against rationality”.

    This is consistent with the simple scenario above and Phil’s scenario, both of which are forms of “you lose if you act rationally”, but there is still a meaningful sense in which rationality always wins. Rationality can be beaten, but only when the game is rigged beforehand so that rationality cannot win.

  • http://profile.typepad.com/1227585392s27146 Philip Goetz

    sternhammer wrote:

    Perhaps you can clarify. Do you, now, believe as you earlier wrote, that:

    “All conservative variants of Christianity teach, in one way or another, that your eternal fate depends on your state in the last moment of your life. If you live a nearly-flawless Christian life, but have a sinful thought ten minutes before dying and the priest has already left, you go to Hell.”

    Try to answer yes or no.

    No, I do not believe that entire sentence. Nor did I originally write that. Eliezer removed some qualifications that I had originally written to that sentence, perhaps because he thought them too wordy, and asked me to check if it was okay. I said that I thought this simplified version would cause people to raise irrelevant objections about purgatory and protestantism, but to go ahead with it anyway.

    I repent of ever writing the original post. 😐 The point I wanted to make was not worth the time we have spent on it, even supposing that it got across.

  • anonymous number 5

    you are forgiven 😛

  • sternhammer

    Thank you.

  • gwern

    “As I’ve posted repeatedly above, I am trying to show that, IF YOU BELIEVE RATIONALITY SHOULD WIN THE NEWCOMB PROBLEM because rationality always wins in real life, THEN you should also believe that it will win in the Augustinian paradox, which, unlike the Newcomb problem, is believed by many people to be a real-life problem. If you’re asking for a real-world, god-free situation (which may be the right thing to do), then you dismiss both paradoxes.”

    Phil: the more I think about it, the more I’m convinced we have to reject any formulation of the paradoxes in which the god or intelligence has certainty* about our decisions or reasons.

    If we imagine that we and the god are both in a simulation, then doesn’t the claim the god can be certain about whether our repentance was planned or ‘genuine’ imply that they (one program) can prove a nontrivial property about another (arbitrary) program? Which would seem to run afoul of Rice’s Theorem/the Halting Problem.

    There are a couple ways out of this that I see:

    1) We can assume the god is or has access to, an oracle. This is explicitly unrealistic though, and I regard such an out as being as bad as saying ‘imagine a universe where 1!=1; Wouldn’t rationality really suck there?’.
    2) We can assume that sentient programs are sufficiently restricted that Rice’s theorem doesn’t apply. That is, any sentient program is by definition restricted enough to be amenable to analysis. I don’t see any a priori reason to believe this, though.
    3) We can suggest that maybe we are provable-about by construction: the god or intelligence created us in such a way that they can prove our unfaithfulness or insincerity, that yes there are beings which the god couldn’t be sure about, but tough luck – we aren’t them and the god knows that. (This is similar to Eliezer’s defense of proofs for FAIness – ‘yeah, we can’t prove friendliness over arbitrary AIs, but we’re not interested in creating arbitrary AIs.’) This doesn’t work for the Newcomb paradox, but could for Augustine’s god.

    * And not just extremely high confidence

  • http://slrman.wordpress.com James Smith

    The optimal strategy is enlightened self-interest.  Always treat others fairly and honestly. That will provide the best long-term results for yourself.  If any god at all exists, you will surely be judged upon your life.  If a god does not, then it isn’t a god worthy of considerations.  Literally, the hell with that one.

  • Muga Sofer

    “I don’t think there’s any rational winning strategy here. Yet the purely emotional strategy of fear plus an irrationally large devaluation of the future wins.”
    No, if you’re good all your life because you want to win you’re still screwed. The only way to win is to coincidentally be an agent that already helps people at cost to itself.

  • dmytryl

    I think in Newcomb’s there’s severe confusion of the aspects that are decision theory and aspects that are the world model. If the predictor works by time travel, you 1-box. If the predictor works by simulation, you also 1-box if your world model is flexible enough to represent a copied instance of a deterministic system (you). If the predictor works by magic, there is a problem that it is not representable in reasonable world models. The canonical predictor works like charisma of the King David, and there’s no actual decision happening, your decision is predetermined.