Score Your Beliefs

A new Games and Economic Behavior paper (ungated here) shows that just asking folks to estimate the chances of events, where they expect to be scored later for accuracy, induces more accurate beliefs about those events, but that what folks believe and say can still consistently diverge:

Belief elicitation in game experiments may be problematic if it changes game play. We experimentally verify that belief elicitation can alter paths of play in a two-player repeated asymmetric matching pennies game. Importantly, this effect occurs only during early periods and only for players with strongly asymmetric payoffs, consistent with a cognitive/affective effect on priors that may serve as a substitute for experience. These effects occur with a common scoring rule elicitation procedure, but not with simpler (unmotivated) statements of expected choices of opponents. Scoring rule belief elicitation improves the goodness of fit of structural models of belief learning, and prior beliefs implied by such models are both stronger and more realistic when beliefs are elicited than when they are not. We also find that “inferred beliefs” (beliefs estimated from past observed actions of opponents) can predict observed actions better than the “stated beliefs” from scoring rule belief elicitation.

Yet more evidence that we should try to get into the habit of collecting track records about our beliefs.

GD Star Rating
Tagged as:
Trackback URL:
  • michael webster

    Robin, I think that you are right about this idea of tracking “records about our beliefs”, especially when we could have rough scores about how right/wrong we were. Do you have any technological suggestions on how to implement this?

    But it does pose a bit of a difficulty in cases in which we don’t have enough similar events, an event space with holes as it were. How would you handle that?

  • Mike

    I hadn’t been a fan of your “track record” recommendations, but I think this is a very good point. I’ve notice in myself, at times, developing an attitude, imagining defending it, imagining being challenged to bet, and then becoming more reserved about it.

    My biggest reservation is that I think there would be far too much noise on any scenario where one might expect this to have value. Consider if we decided to use a national NCAA mens basketball bracket tournament to determine who should be the sports commentators. I believe there is some understanding one can bring to predicting basketball game outcomes, and combined with deep knowledge of the teams, this should make one a better analysis of sporting events. But these people will rarely come out at the top of the bracket tournament. Bracket tournaments are often won by people who make unusual predictions based on biased loyalties, which by chance happen to play out in a given year. Other people win by guesswork. There is so much chance in a basketball tournament, that I’m not convinced even repeating this program year after year will reveal who is most likely to perform among the best in the next year’s tournament.

    The cost of this noise is false confidence — people claiming to have the best advice because of their track record, when in fact they were just lucky.

  • Doug S.

    The cost of this noise is false confidence — people claiming to have the best advice because of their track record, when in fact they were just lucky.

    Definitely… consider actively managed stock funds; some money managers get lucky and beat the market, but past performance is generally uncorrelated with future results.

  • Dagon

    Track records would be nice, but just getting in the habit of assigning (and requesting others to give) probability estimates to beliefs, rather than binaries, would probably get us a lot of benefit.

  • Stephen

    “The cost of this noise is false confidence — people claiming to have the best advice because of their track record, when in fact they were just lucky.”

    No reason you couldn’t start a track record of track records to determine which events are reliably predicted by track records.

    Nobody would suggest a prediction market for lottery numbers.

  • matt

    Try as a way of keeping track.
    (We’re still in closed beta, but you’re all close friends here. We’ll go live at when we’re ready. Any history you generate before we go live will be preserved.)

    Notes & disclaimers:

    • If this app is wildly successful, I may make money from it – I hope you’ll agree that this isn’t spam because I’m an avid reader of OB, we wrote this app after reading OB and related material convinced me that it should exist, and I genuinely believe that some of you will like it.
    • We’re still in “closed” beta, and there are some rough edges still to be worked off (in particular, if you’re using an Internet Explorer, consider using a good browser instead – we’re still fighting with IE and don’t expect to be finished for at least a couple of weeks).
    • We’re very interested in your feedback – please use the feedback tab on the right of the site.
  • Robin Hanson

    Matt, very cool! 🙂

  • Pingback: PredictionBook | Midas Oracle .ORG