How Spend Rationality Test?

To learn to be more (epistemically) rational,  i.e., to better discern and tell the truth, it would sure help to have good ways to regularly test ourselves.  Eliezer and I have both requested folks to give thought to how we could better test our (epistemic) rationality.

It seems relatively easy to test someone's ability to make accurate and calibrated forecasts in novel contexts.  Given them some info on a new topic, limited time and resources to make estimates, and then evaluate their accuracy.  We might presume that any residual after controlling for IQ, info, effort, and related expertise was (epistemic) "rationality." 

My main worry about this approach is it doesn't get at the fact that some topics test one's rationality more severely than others.  It can be much harder to be honest when you care a lot about a topic, when others care about your opinion, or when you don't expect your opinions to be scored against reality anytime soon. How can we test rationality in these cases?

Over twenty years ago some psychologists worked out a twenty item questionnaire for evaluating how much people lie to themselves to look good, on topics important to them.  (A related twenty question survey looks at lies to others.)  They "validated" these "scales" by running a lot of tests comparing them to other scales. 

Alas, I doubt that these tests would work as well if respondents knew that they were being tested for rationality.  And surely couldn't tell someone their score and then give them the test again and expect it to be as informative.  So these tests are a valuable but limited resource.

Which is why I haven't linked to them here in this post, yet.  First I want folks to ponder:  how best could we spend this limited resource to test our rationality?

Added 1Apr: The test is in here; don't look unless you've thought about whether waiting might be better.

GD Star Rating
loading...
Tagged as:
Trackback URL:
  • http://macroethics.blogspot.com nazgulnarsil

    wait until everyone forgets about this post and then post it disguised as something completely different.

  • Jayson Virissimo

    IDK, we could like… bet on stuff.

  • Kenny Evitt

    I don’t think your main worry is sufficient reason not to play with the idea. I can imagine a very workable web-app that would:

    1. Email nicknamed users “some info on a new topic”; it could (should?) be anything – news stories, research papers, submitted tests, etc.
    2. And provide limited time “to make estimates” (and submit them); they could be limited to a specific set of fixed statements, or free-form

    Such a mound of data would be fun to analyze, even accounting for the uncontrolled experimental setting. Looking at specific topic categories would make up for “… the fact that some topics test one’s rationality more severely than others.”

    Or maybe you could require estimates to post comments on selected topics.

  • http://yudkowsky.net/ Eliezer Yudkowsky

    I don’t see any tremendously good use available right now. I’d just as soon save something like this until better institutions were or began to be set up.

  • Stuart Armstrong

    First I want folks to ponder: how best could we spend this limited resource to test our rationality?

    Is it an intelligent use of our time to do so? We all agree that there is not enough rationality in the world, that there should be more. In that case, why not just use some cheap scale that can show a relative change in rationality, rather than worrying about the absolute amount?

    When we percieve the world becoming saturated with rationality, then maybe we need to worry about accurate absolute measures.

  • Seb

    >My main worry about this approach is it doesn’t get at the fact that some topics test one’s rationality more severely than others. It can be much harder to be honest when you care a lot about a topic, when others care about your opinion, or when you don’t expect your opinions to be scored against reality anytime soon. How can we test rationality in these cases?

    my take on this is: we can test rationality in these cases just as easily as in any other case, and what we would then surely find is that “in these cases” people are less rational.

    if i understand you correctly, you seem to be assuming that there is such a thing as “how rational a person ACTUALLY is”, but that when their thinking is muddled by phenomena of social psychology their “actual” rationality doesn’t quite show – ? i’m not sure i understand you correctly, because that assumption seems far fetched to me. why would there be a fixed, “actual” rationality?

  • Matthew

    Could someone give an example of what is meant by a reversal test? I don’t want to spoil the scoring system for anyone that takes it however. My score is adding up to a negative number which isnt on the scale.

  • http://profile.typepad.com/robinhanson Robin Hanson

    Stuart, a cheap scale for showing rationality change wold be welcome.

    Seb, I didn’t mean to imply we are equally rational on all topics.

  • Cyan

    How Spend Rationality Test?

    Buy pronoun?

  • frelkins

    @Cyan

    Why pronoun?

    All he needs is the marker to.

  • Curt Adams

    I don’t think there’s any great reason to put off the test. Mostly I found it raised philosophical issues when I was “self-deceptive”, like the meanings of “sometimes” and “always” and the difference between intent and execution. It’s an interesting read but probably won’t give any great revelations to a reader of this blog.

    I though it was interesting that I’m a “nice guy” even though I give a lot of not-nice answers (do people lie, do I lie, do I get mad, etc.) I’m thinking of adopting and it makes me wonder what I should teach a child about morality. Is it better to teach the “standard way” that lying, cheating, etc. is wrong and have the child self-deceive to do it when appropriate? Or is is it better to teach the “cynical nice guy” approach I’ve come around to, which is that in the intensely social and cooperative world humans inhabit, you should be “nice” most of the time because that’s how you get along and get ahead?

  • http://yudkowsky.net/ Eliezer Yudkowsky

    Spend them on monitoring people under fMRIs to see if we can detect neural correlates of rationalization and the like.

  • Anonymous

    The link doesn’t appear to be working- could somebody refer me to the actual test?