Eliezer once wrote this about Newcomb’s problem: Nonetheless, I would like to present some of my motivations on Newcomb’s Problem – the reasons I felt impelled to seek a new theory – because they illustrate my source-attitudes toward rationality. Even if I can’t present the theory that these motivations motivate…
Well, suppose I make a copy of you, and give it $1000 , and if it declines, I give real you $10000 . Which I don't tell to the copy of you. Now some sort of agent that declines $1000, but takes $10000 (being overwhelmed by greed for example) wins, and nobody needs to look at anyone's source code in any detail, everything can be black boxed.
I think in Newcomb's there's severe confusion of aspects that are part of decision theory and aspects that belong in the world model. If the predictor works by time travel, you 1-box (One could implement a world with time travel where software would find a stable solution). If the predictor works by simulation, you also 1-box if your world model is flexible enough to represent a copied instance of any deterministic system (including you), otherwise you may 2-box but chiefly because you can't represent the predictor correctly - the agent is expecting, on the formal level, that it is getting unknown+1000 , a clear cut case of failure to even predict what you are getting let alone make a choice.
If the predictor works by magic, there is a problem that it is not representable in reasonable world models. The canonical predictor works like charisma of the King David, and there's no actual decision happening, your decision is predetermined.
It is all a lot clearer from the perspective of writing some simple practical AI that models the world, tries it's actions in the world, and decides.
"I don't think there's any rational winning strategy here. Yet the purely emotional strategy of fear plus an irrationally large devaluation of the future wins."No, if you're good all your life because you want to win you're still screwed. The only way to win is to coincidentally be an agent that already helps people at cost to itself.
The optimal strategy is enlightened self-interest. Always treat others fairly and honestly. That will provide the best long-term results for yourself. If any god at all exists, you will surely be judged upon your life. If a god does not, then it isn't a god worthy of considerations. Literally, the hell with that one.
"As I've posted repeatedly above, I am trying to show that, IF YOU BELIEVE RATIONALITY SHOULD WIN THE NEWCOMB PROBLEM because rationality always wins in real life, THEN you should also believe that it will win in the Augustinian paradox, which, unlike the Newcomb problem, is believed by many people to be a real-life problem. If you're asking for a real-world, god-free situation (which may be the right thing to do), then you dismiss both paradoxes."
Phil: the more I think about it, the more I'm convinced we have to reject any formulation of the paradoxes in which the god or intelligence has certainty* about our decisions or reasons.
If we imagine that we and the god are both in a simulation, then doesn't the claim the god can be certain about whether our repentance was planned or 'genuine' imply that they (one program) can prove a nontrivial property about another (arbitrary) program? Which would seem to run afoul of Rice's Theorem/the Halting Problem.
There are a couple ways out of this that I see:
1) We can assume the god is or has access to, an oracle. This is explicitly unrealistic though, and I regard such an out as being as bad as saying 'imagine a universe where 1!=1; Wouldn't rationality really suck there?'. 2) We can assume that sentient programs are sufficiently restricted that Rice's theorem doesn't apply. That is, any sentient program is by definition restricted enough to be amenable to analysis. I don't see any a priori reason to believe this, though.3) We can suggest that maybe we are provable-about by construction: the god or intelligence created us in such a way that they can prove our unfaithfulness or insincerity, that yes there are beings which the god couldn't be sure about, but tough luck - we aren't them and the god knows that. (This is similar to Eliezer's defense of proofs for FAIness - 'yeah, we can't prove friendliness over arbitrary AIs, but we're not interested in creating arbitrary AIs.') This doesn't work for the Newcomb paradox, but could for Augustine's god.
Perhaps you can clarify. Do you, now, believe as you earlier wrote, that:
"All conservative variants of Christianity teach, in one way or another, that your eternal fate depends on your state in the last moment of your life. If you live a nearly-flawless Christian life, but have a sinful thought ten minutes before dying and the priest has already left, you go to Hell."
Try to answer yes or no.
No, I do not believe that entire sentence. Nor did I originally write that. Eliezer removed some qualifications that I had originally written to that sentence, perhaps because he thought them too wordy, and asked me to check if it was okay. I said that I thought this simplified version would cause people to raise irrelevant objections about purgatory and protestantism, but to go ahead with it anyway.
I repent of ever writing the original post. :| The point I wanted to make was not worth the time we have spent on it, even supposing that it got across.
The details of planning, sinning, and Christianity are all irrelevant to the underlying point. I think the following is a simpler version of Phil's scenario that illustrates the point.
Let us suppose that God decides who goes to heaven and who goes to hell using the following rule: you go to heaven if and only if you have never been rational with respect to heaven or hell, otherwise you go to hell.
Obviously, a rational agent always loses this game, and non-rational agents always win.
I think that what scenarios like this show is that you can't say just "rationality always wins", or more carefully, "no strategy has a better expected outcome than rationality". Instead, you have to say that "no rationally chosen strategy has a better expected outcome than rationality", or "no strategy has a better expected outcome than rationality unless the game includes an a priori penalty against rationality".
This is consistent with the simple scenario above and Phil's scenario, both of which are forms of "you lose if you act rationally", but there is still a meaningful sense in which rationality always wins. Rationality can be beaten, but only when the game is rigged beforehand so that rationality cannot win.
- Intuitively insightful fools. Agents with no conscious attempt at rationality, but possessing an intuitive, meta-level understanding of the behaviors of other agents in their social environment, sufficient to work within the environment to obtain more optimal results.
Do you meant to suggest here that rationality requires consciousness or that an intuitive process that was provably optimal in all circumstances would not be a rational process? It sounds like it. If so, this seems problematic, as I hope the following thought experiment shows.
Consider two people whose brains are working on a complex problem (that doesn't require and isn't aided by consciousness of any form), with the only relevant difference between what the 2 brains are doing being that one has conscious awareness of (some of) what it is doing as it works on the problem. (If you think this is impossible in principle, please say why.) If they both reliably solve the problem optimally -- there is no other strategy of any kind that is superior to either one -- via an analogous sequence of steps and calculations, on what basis can you say that one is rational and the other is not?
I don't think intuition ever gets this reliable, and so it is not rational for that reason, but if it could be so reliable (which seems possible in principle for domains that don't involve consciousness [e.g., anything a computer could do optimally], although even then, perhaps consciousness could be modeled 'intuitively' in a non-conscious manner), then it would be a form of rationality.
What would you call a state in which you have internalized the principles and rules for a domain of problems that you can now solve instantly without any conscious effort whatsoever? It sounds like a form of intuition to me. And yet, it's hard to say that the process was a form of rationality for the entire time that it required effort and was conscious, but that after you became so skilled at the intellectual tasks that they just began to happen automatically and effortlessly, the process suddenly became non-rational.
Nick Tarleton,how is PD like Newcomb?Newcomb seems much easier to me than PD.
Phil Goetz,Lots of people responded before your hysterical outburst. I would have, too, if sternhammer hadn't hijacked the thread. Eliezer, in the very first comment, lays out his position quite clearly. In particular, he hints that he does believe that "A rational agent should get the maximum payoff in the PD." The absolute max is impossible from symmetry, but he has said elsewhere on OB that two transparent rational agents cooperate in a one-shot PD.
You seem to be upset that he didn't address the meaning of the slogan "rational agents should win." This is stupid to argue about. I endorse Nick Tarleton's version.
I disagreed with one particular sentence you wrote, quoting it several times. At no point were you able to admit that you were wrong about that. I think you are hinting that you now know it is wrong. But I am not sure. Your post seems very oblique.
Perhaps you can clarify. Do you, now, believe as you earlier wrote, that:
"All conservative variants of Christianity teach, in one way or another, that your eternal fate depends on your state in the last moment of your life. If you live a nearly-flawless Christian life, but have a sinful thought ten minutes before dying and the priest has already left, you go to Hell."
Try to answer yes or no.
If no, then we have no argument.
I have read your last paragraph many times without understanding what you are saying. Please edit it for clarity. Are you talking about something I have claimed or something you have claimed?
And when you reject my proposed test as biased, despite my offering you the exact language I would use and your alleging no bias in that, perhaps you would consider offering a test of your own.
But not offering a test, and not offering supporting evidence that can be checked, would be very consistent with an effort to avoid detection of errors in your claims. I can't read your mind. But I spent years as a lawyer, and when a witness acts like that, it speaks louder than anything he says.
No, I don't want you bothering my ex-pastors. And I know perfectly well that you can get different answers from them depending on how you ask the question. I will not bet money on their rationality and logical consistence.
I already wrote, "The second part, that you go to heaven if you sincerely repent (and meet other conditions that may include confession, baptism, etc.) before dying, is taught by almost every Christian church, conservative, Catholic, Protestant, or otherwise. (The major exception is forms of predestination that teach that you can never know whether you are saved or not.) That is the only part that matters for this discussion."
I also explained some of the reasons behind my opinions on particular Catholic beliefs. And I referred, twice, to the episode of the thief on the cross as the explanation of why all Christians believe you can be saved at the end of your life regardless of how you have lived your life before that.
You seem unaware that what you are claiming, in light of what I have said, is that most Christian churches teach that you can still go to Hell even after you have sincerely repented. You keep repeating my first statement and ignoring all of my later clarifications and qualifications. You keep asking for responses I have already made. I can only conclude you have written a long impassioned response to me without reading what I said.
(i posted this... and it showed up. but the comments were different. then i signed in under my typekey username and my comment went away. so i'm reposting it. sorry if it got lost, or posted twice.)
Phil Goetz: "Yes. But I'm conducting a thought experiment. I don't mean it as an attack on Christianity. Tho I can see how it could be seen that way. "
i didn't read your post as an attack on Christianity (though i did pickup that you don't consider yourself Christian any longer). My point was that, the thought experiment, the paradox, as you phrased it was, "The optimal self-interested strategy is to act selfishly all your life, and then repent at the final moment" and that is wrong.
The optimal self-interested strategy isn't to act selfishly all your life, unless the end goal is simply avoiding hell. If your end goal is maximizing enjoyment in life, then living a righteous life is the optimal self-interested strategy.
the rest of your argument relied on that statement. and it is false, rationally speaking.
Sternhammer:
I get what you're trying to say, but i believe you misread Goetz. When Goetz saed, "All conservative variants of Christianity teach, in one way or another, that your eternal fate depends on your state in the last moment of your life", thats true. his followup statements, "If you live a nearly-flawless Christian life, but have a sinful thought ten minutes before dying and the priest has already left, you go to Hell. If you are sinful all your life but repent in your final minute, you go to Heaven", should be read as an either/or. Some evangelicals believe you can loose your salvation (typically you find this in the more charismatic denominations), just as he described in his first statement.
Either way, your fate does depend on your "state" in the last moment of your life. even if that "state" was decided 80 years ago or 5 minutes ago. Its true for the "once saved, always saved" crowd.
Phil Goetz: "Yes. But I'm conducting a thought experiment. I don't mean it as an attack on Christianity. Tho I can see how it could be seen that way. "
i didn't read your post as an attack on Christianity (though i did pickup that you don't consider yourself Christian any longer). My point was that, the thought experiment, the paradox, as you phrased it was, "The optimal self-interested strategy is to act selfishly all your life, and then repent at the final moment" and that is wrong.
The optimal self-interested strategy isn't to act selfishly all your life, unless the end goal is simply avoiding hell. If your end goal is maximizing enjoyment in life, then living a righteous life is the optimal self-interested strategy.
the rest of your argument relied on that statement. and it is false, rationally speaking.
Sternhammer:
I get what you're trying to say, but i believe you misread Goetz. When Goetz saed, "All conservative variants of Christianity teach, in one way or another, that your eternal fate depends on your state in the last moment of your life", thats true. his followup statements, "If you live a nearly-flawless Christian life, but have a sinful thought ten minutes before dying and the priest has already left, you go to Hell. If you are sinful all your life but repent in your final minute, you go to Heaven", should be read as an either/or. Some evangelicals believe you can loose your salvation (typically you find this in the more charismatic denominations), just as he described in his first statement.
Either way, your fate does depend on your "state" in the last moment of your life. even if that "state" was decided 80 years ago or 5 minutes ago. Its true for the "once saved, always saved" crowd.
Well, suppose I make a copy of you, and give it $1000 , and if it declines, I give real you $10000 . Which I don't tell to the copy of you. Now some sort of agent that declines $1000, but takes $10000 (being overwhelmed by greed for example) wins, and nobody needs to look at anyone's source code in any detail, everything can be black boxed.
I think in Newcomb's there's severe confusion of aspects that are part of decision theory and aspects that belong in the world model. If the predictor works by time travel, you 1-box (One could implement a world with time travel where software would find a stable solution). If the predictor works by simulation, you also 1-box if your world model is flexible enough to represent a copied instance of any deterministic system (including you), otherwise you may 2-box but chiefly because you can't represent the predictor correctly - the agent is expecting, on the formal level, that it is getting unknown+1000 , a clear cut case of failure to even predict what you are getting let alone make a choice.
If the predictor works by magic, there is a problem that it is not representable in reasonable world models. The canonical predictor works like charisma of the King David, and there's no actual decision happening, your decision is predetermined.
It is all a lot clearer from the perspective of writing some simple practical AI that models the world, tries it's actions in the world, and decides.
"I don't think there's any rational winning strategy here. Yet the purely emotional strategy of fear plus an irrationally large devaluation of the future wins."No, if you're good all your life because you want to win you're still screwed. The only way to win is to coincidentally be an agent that already helps people at cost to itself.
The optimal strategy is enlightened self-interest. Always treat others fairly and honestly. That will provide the best long-term results for yourself. If any god at all exists, you will surely be judged upon your life. If a god does not, then it isn't a god worthy of considerations. Literally, the hell with that one.
"As I've posted repeatedly above, I am trying to show that, IF YOU BELIEVE RATIONALITY SHOULD WIN THE NEWCOMB PROBLEM because rationality always wins in real life, THEN you should also believe that it will win in the Augustinian paradox, which, unlike the Newcomb problem, is believed by many people to be a real-life problem. If you're asking for a real-world, god-free situation (which may be the right thing to do), then you dismiss both paradoxes."
Phil: the more I think about it, the more I'm convinced we have to reject any formulation of the paradoxes in which the god or intelligence has certainty* about our decisions or reasons.
If we imagine that we and the god are both in a simulation, then doesn't the claim the god can be certain about whether our repentance was planned or 'genuine' imply that they (one program) can prove a nontrivial property about another (arbitrary) program? Which would seem to run afoul of Rice's Theorem/the Halting Problem.
There are a couple ways out of this that I see:
1) We can assume the god is or has access to, an oracle. This is explicitly unrealistic though, and I regard such an out as being as bad as saying 'imagine a universe where 1!=1; Wouldn't rationality really suck there?'. 2) We can assume that sentient programs are sufficiently restricted that Rice's theorem doesn't apply. That is, any sentient program is by definition restricted enough to be amenable to analysis. I don't see any a priori reason to believe this, though.3) We can suggest that maybe we are provable-about by construction: the god or intelligence created us in such a way that they can prove our unfaithfulness or insincerity, that yes there are beings which the god couldn't be sure about, but tough luck - we aren't them and the god knows that. (This is similar to Eliezer's defense of proofs for FAIness - 'yeah, we can't prove friendliness over arbitrary AIs, but we're not interested in creating arbitrary AIs.') This doesn't work for the Newcomb paradox, but could for Augustine's god.
* And not just extremely high confidence
Thank you.
you are forgiven :P
sternhammer wrote:
Perhaps you can clarify. Do you, now, believe as you earlier wrote, that:
"All conservative variants of Christianity teach, in one way or another, that your eternal fate depends on your state in the last moment of your life. If you live a nearly-flawless Christian life, but have a sinful thought ten minutes before dying and the priest has already left, you go to Hell."
Try to answer yes or no.
No, I do not believe that entire sentence. Nor did I originally write that. Eliezer removed some qualifications that I had originally written to that sentence, perhaps because he thought them too wordy, and asked me to check if it was okay. I said that I thought this simplified version would cause people to raise irrelevant objections about purgatory and protestantism, but to go ahead with it anyway.
I repent of ever writing the original post. :| The point I wanted to make was not worth the time we have spent on it, even supposing that it got across.
The details of planning, sinning, and Christianity are all irrelevant to the underlying point. I think the following is a simpler version of Phil's scenario that illustrates the point.
Let us suppose that God decides who goes to heaven and who goes to hell using the following rule: you go to heaven if and only if you have never been rational with respect to heaven or hell, otherwise you go to hell.
Obviously, a rational agent always loses this game, and non-rational agents always win.
I think that what scenarios like this show is that you can't say just "rationality always wins", or more carefully, "no strategy has a better expected outcome than rationality". Instead, you have to say that "no rationally chosen strategy has a better expected outcome than rationality", or "no strategy has a better expected outcome than rationality unless the game includes an a priori penalty against rationality".
This is consistent with the simple scenario above and Phil's scenario, both of which are forms of "you lose if you act rationally", but there is still a meaningful sense in which rationality always wins. Rationality can be beaten, but only when the game is rigged beforehand so that rationality cannot win.
@soulless:
- Intuitively insightful fools. Agents with no conscious attempt at rationality, but possessing an intuitive, meta-level understanding of the behaviors of other agents in their social environment, sufficient to work within the environment to obtain more optimal results.
Do you meant to suggest here that rationality requires consciousness or that an intuitive process that was provably optimal in all circumstances would not be a rational process? It sounds like it. If so, this seems problematic, as I hope the following thought experiment shows.
Consider two people whose brains are working on a complex problem (that doesn't require and isn't aided by consciousness of any form), with the only relevant difference between what the 2 brains are doing being that one has conscious awareness of (some of) what it is doing as it works on the problem. (If you think this is impossible in principle, please say why.) If they both reliably solve the problem optimally -- there is no other strategy of any kind that is superior to either one -- via an analogous sequence of steps and calculations, on what basis can you say that one is rational and the other is not?
I don't think intuition ever gets this reliable, and so it is not rational for that reason, but if it could be so reliable (which seems possible in principle for domains that don't involve consciousness [e.g., anything a computer could do optimally], although even then, perhaps consciousness could be modeled 'intuitively' in a non-conscious manner), then it would be a form of rationality.
What would you call a state in which you have internalized the principles and rules for a domain of problems that you can now solve instantly without any conscious effort whatsoever? It sounds like a form of intuition to me. And yet, it's hard to say that the process was a form of rationality for the entire time that it required effort and was conscious, but that after you became so skilled at the intellectual tasks that they just began to happen automatically and effortlessly, the process suddenly became non-rational.
I disagreed with one particular sentence you wrote, quoting it several times. At no point were you able to admit that you were wrong about that.
sternhammer:Phil::Phil:Eliezer
Nick Tarleton,how is PD like Newcomb?Newcomb seems much easier to me than PD.
Phil Goetz,Lots of people responded before your hysterical outburst. I would have, too, if sternhammer hadn't hijacked the thread. Eliezer, in the very first comment, lays out his position quite clearly. In particular, he hints that he does believe that "A rational agent should get the maximum payoff in the PD." The absolute max is impossible from symmetry, but he has said elsewhere on OB that two transparent rational agents cooperate in a one-shot PD.
You seem to be upset that he didn't address the meaning of the slogan "rational agents should win." This is stupid to argue about. I endorse Nick Tarleton's version.
Phil,
I disagreed with one particular sentence you wrote, quoting it several times. At no point were you able to admit that you were wrong about that. I think you are hinting that you now know it is wrong. But I am not sure. Your post seems very oblique.
Perhaps you can clarify. Do you, now, believe as you earlier wrote, that:
"All conservative variants of Christianity teach, in one way or another, that your eternal fate depends on your state in the last moment of your life. If you live a nearly-flawless Christian life, but have a sinful thought ten minutes before dying and the priest has already left, you go to Hell."
Try to answer yes or no.
If no, then we have no argument.
I have read your last paragraph many times without understanding what you are saying. Please edit it for clarity. Are you talking about something I have claimed or something you have claimed?
And when you reject my proposed test as biased, despite my offering you the exact language I would use and your alleging no bias in that, perhaps you would consider offering a test of your own.
But not offering a test, and not offering supporting evidence that can be checked, would be very consistent with an effort to avoid detection of errors in your claims. I can't read your mind. But I spent years as a lawyer, and when a witness acts like that, it speaks louder than anything he says.
sternhammer,
No, I don't want you bothering my ex-pastors. And I know perfectly well that you can get different answers from them depending on how you ask the question. I will not bet money on their rationality and logical consistence.
I already wrote, "The second part, that you go to heaven if you sincerely repent (and meet other conditions that may include confession, baptism, etc.) before dying, is taught by almost every Christian church, conservative, Catholic, Protestant, or otherwise. (The major exception is forms of predestination that teach that you can never know whether you are saved or not.) That is the only part that matters for this discussion."
I also explained some of the reasons behind my opinions on particular Catholic beliefs. And I referred, twice, to the episode of the thief on the cross as the explanation of why all Christians believe you can be saved at the end of your life regardless of how you have lived your life before that.
You seem unaware that what you are claiming, in light of what I have said, is that most Christian churches teach that you can still go to Hell even after you have sincerely repented. You keep repeating my first statement and ignoring all of my later clarifications and qualifications. You keep asking for responses I have already made. I can only conclude you have written a long impassioned response to me without reading what I said.
(i posted this... and it showed up. but the comments were different. then i signed in under my typekey username and my comment went away. so i'm reposting it. sorry if it got lost, or posted twice.)
Phil Goetz: "Yes. But I'm conducting a thought experiment. I don't mean it as an attack on Christianity. Tho I can see how it could be seen that way. "
i didn't read your post as an attack on Christianity (though i did pickup that you don't consider yourself Christian any longer). My point was that, the thought experiment, the paradox, as you phrased it was, "The optimal self-interested strategy is to act selfishly all your life, and then repent at the final moment" and that is wrong.
The optimal self-interested strategy isn't to act selfishly all your life, unless the end goal is simply avoiding hell. If your end goal is maximizing enjoyment in life, then living a righteous life is the optimal self-interested strategy.
the rest of your argument relied on that statement. and it is false, rationally speaking.
Sternhammer:
I get what you're trying to say, but i believe you misread Goetz. When Goetz saed, "All conservative variants of Christianity teach, in one way or another, that your eternal fate depends on your state in the last moment of your life", thats true. his followup statements, "If you live a nearly-flawless Christian life, but have a sinful thought ten minutes before dying and the priest has already left, you go to Hell. If you are sinful all your life but repent in your final minute, you go to Heaven", should be read as an either/or. Some evangelicals believe you can loose your salvation (typically you find this in the more charismatic denominations), just as he described in his first statement.
Either way, your fate does depend on your "state" in the last moment of your life. even if that "state" was decided 80 years ago or 5 minutes ago. Its true for the "once saved, always saved" crowd.
Phil Goetz: "Yes. But I'm conducting a thought experiment. I don't mean it as an attack on Christianity. Tho I can see how it could be seen that way. "
i didn't read your post as an attack on Christianity (though i did pickup that you don't consider yourself Christian any longer). My point was that, the thought experiment, the paradox, as you phrased it was, "The optimal self-interested strategy is to act selfishly all your life, and then repent at the final moment" and that is wrong.
The optimal self-interested strategy isn't to act selfishly all your life, unless the end goal is simply avoiding hell. If your end goal is maximizing enjoyment in life, then living a righteous life is the optimal self-interested strategy.
the rest of your argument relied on that statement. and it is false, rationally speaking.
Sternhammer:
I get what you're trying to say, but i believe you misread Goetz. When Goetz saed, "All conservative variants of Christianity teach, in one way or another, that your eternal fate depends on your state in the last moment of your life", thats true. his followup statements, "If you live a nearly-flawless Christian life, but have a sinful thought ten minutes before dying and the priest has already left, you go to Hell. If you are sinful all your life but repent in your final minute, you go to Heaven", should be read as an either/or. Some evangelicals believe you can loose your salvation (typically you find this in the more charismatic denominations), just as he described in his first statement.
Either way, your fate does depend on your "state" in the last moment of your life. even if that "state" was decided 80 years ago or 5 minutes ago. Its true for the "once saved, always saved" crowd.