A new Games and Economic Behavior paper (ungated here) shows that just asking folks to estimate the chances of events, where they expect to be scored later for accuracy, induces more accurate beliefs about those events, but that what folks believe and say can still consistently diverge:
Try http://pbook.trike.com.au/ as a way of keeping track.(We're still in closed beta, but you're all close friends here. We'll go live at predictionbook.com when we're ready. Any history you generate before we go live will be preserved.)
Notes & disclaimers:<ul><li>If this app is wildly successful, I may make money from it - I hope you'll agree that this isn't spam because I'm an avid reader of OB, we wrote this app after reading OB and related material convinced me that it should exist, and I genuinely believe that some of you will like it.</li><li>We're still in "closed" beta, and there are some rough edges still to be worked off (in particular, if you're using an Internet Explorer, consider using a good browser instead - we're still fighting with IE and don't expect to be finished for at least a couple of weeks).</li><li>We're very interested in your feedback - please use the feedback tab on the right of the site.</li></ul>
"The cost of this noise is false confidence -- people claiming to have the best advice because of their track record, when in fact they were just lucky."
No reason you couldn't start a track record of track records to determine which events are reliably predicted by track records.
Nobody would suggest a prediction market for lottery numbers.
Track records would be nice, but just getting in the habit of assigning (and requesting others to give) probability estimates to beliefs, rather than binaries, would probably get us a lot of benefit.
The cost of this noise is false confidence -- people claiming to have the best advice because of their track record, when in fact they were just lucky.
Definitely... consider actively managed stock funds; some money managers get lucky and beat the market, but past performance is generally uncorrelated with future results.
I hadn't been a fan of your "track record" recommendations, but I think this is a very good point. I've notice in myself, at times, developing an attitude, imagining defending it, imagining being challenged to bet, and then becoming more reserved about it.
My biggest reservation is that I think there would be far too much noise on any scenario where one might expect this to have value. Consider if we decided to use a national NCAA mens basketball bracket tournament to determine who should be the sports commentators. I believe there is some understanding one can bring to predicting basketball game outcomes, and combined with deep knowledge of the teams, this should make one a better analysis of sporting events. But these people will rarely come out at the top of the bracket tournament. Bracket tournaments are often won by people who make unusual predictions based on biased loyalties, which by chance happen to play out in a given year. Other people win by guesswork. There is so much chance in a basketball tournament, that I'm not convinced even repeating this program year after year will reveal who is most likely to perform among the best in the next year's tournament.
The cost of this noise is false confidence -- people claiming to have the best advice because of their track record, when in fact they were just lucky.
Robin, I think that you are right about this idea of tracking "records about our beliefs", especially when we could have rough scores about how right/wrong we were. Do you have any technological suggestions on how to implement this?
But it does pose a bit of a difficulty in cases in which we don't have enough similar events, an event space with holes as it were. How would you handle that?
Matt, very cool! :)
Try http://pbook.trike.com.au/ as a way of keeping track.(We're still in closed beta, but you're all close friends here. We'll go live at predictionbook.com when we're ready. Any history you generate before we go live will be preserved.)
Notes & disclaimers:<ul><li>If this app is wildly successful, I may make money from it - I hope you'll agree that this isn't spam because I'm an avid reader of OB, we wrote this app after reading OB and related material convinced me that it should exist, and I genuinely believe that some of you will like it.</li><li>We're still in "closed" beta, and there are some rough edges still to be worked off (in particular, if you're using an Internet Explorer, consider using a good browser instead - we're still fighting with IE and don't expect to be finished for at least a couple of weeks).</li><li>We're very interested in your feedback - please use the feedback tab on the right of the site.</li></ul>
"The cost of this noise is false confidence -- people claiming to have the best advice because of their track record, when in fact they were just lucky."
No reason you couldn't start a track record of track records to determine which events are reliably predicted by track records.
Nobody would suggest a prediction market for lottery numbers.
Track records would be nice, but just getting in the habit of assigning (and requesting others to give) probability estimates to beliefs, rather than binaries, would probably get us a lot of benefit.
The cost of this noise is false confidence -- people claiming to have the best advice because of their track record, when in fact they were just lucky.
Definitely... consider actively managed stock funds; some money managers get lucky and beat the market, but past performance is generally uncorrelated with future results.
I hadn't been a fan of your "track record" recommendations, but I think this is a very good point. I've notice in myself, at times, developing an attitude, imagining defending it, imagining being challenged to bet, and then becoming more reserved about it.
My biggest reservation is that I think there would be far too much noise on any scenario where one might expect this to have value. Consider if we decided to use a national NCAA mens basketball bracket tournament to determine who should be the sports commentators. I believe there is some understanding one can bring to predicting basketball game outcomes, and combined with deep knowledge of the teams, this should make one a better analysis of sporting events. But these people will rarely come out at the top of the bracket tournament. Bracket tournaments are often won by people who make unusual predictions based on biased loyalties, which by chance happen to play out in a given year. Other people win by guesswork. There is so much chance in a basketball tournament, that I'm not convinced even repeating this program year after year will reveal who is most likely to perform among the best in the next year's tournament.
The cost of this noise is false confidence -- people claiming to have the best advice because of their track record, when in fact they were just lucky.
Robin, I think that you are right about this idea of tracking "records about our beliefs", especially when we could have rough scores about how right/wrong we were. Do you have any technological suggestions on how to implement this?
But it does pose a bit of a difficulty in cases in which we don't have enough similar events, an event space with holes as it were. How would you handle that?