The 80% Forecasting Solution
We often have remarkably poor forecasts, relative to the information around. An exciting new way to do better is prediction markets; have people bet on your topic. But honestly, I’d guess you can get 80% of the improvement that predict markets offer by using a much simpler solution: collect track records.
Yup, it’s that simple. When people make forecast-like-statements, write them down in a clear standardized form, and then check back later to see who was more accurate. Along the way, create a consensus forecast by averaging recent forecasts, perhaps weighted by accuracy or expert credentials. If you collect enough forecasts to evaluate accuracy, and reward accuracy well enough, people will try hard to be right, and you’ll learn what kinds of people to listen to.
Now don’t get me wrong; prediction markets offer real advantages over simple track records. Prediction markets can directly pay people with short track records for adding information. They don’t require you to decide who is more accurate when, and they are probably updated more quickly. On the other hand, complex trading environments discourage many people from contributing their information. So some vendors, such as Newsfutures and HP, are experimenting with skipping the trading and just getting people to make comparable forecasts.
It is something of a puzzle that we don’t demand more track records from our advisers. If you go to a cancer doctor, for example, he will not show you a record of his previous patients and how long they lived after his treatments. In fact, he usually has no such records, as he has typically lost track of his old patients. Similarly for the guy who repairs your car, or who treats your lawn, or who teaches your classes.
Yes, it would be trouble to collect and evaluate track records. But if customers cared enough, records would happen. Consider that companies go through the bother of hiring auditors to check financial records because investors insist on it; why don’t we insist on track records? Until we understand why people haven’t opted for the 80% solution of track records, I’m afraid we can’t be that optimistic about prediction markets either.
P.S. Today I’m presenting at a Yahoo prediction markets meeting.
Hal, it is hard to sympathize with people whose job it is to make forecasts, such as cancer doctors, who are reluctant to be "put on the spot."
One thing to watch out for is that many people are reluctant to be put on the spot by having their forecasts recorded and judged. Tetlock ran into this with his study, he had to offer his subjects anonymity. (And it's a good thing for them he did, since they did so badly.)
The problem is that it might be that some of the best forecasters are most uncertain about their forecasts (Tetlock's foxes) and therefore might be reluctant to be scored like this, while some of the worst forecasters are overconfident (hedgehogs) and would be overrepresented in the sample. While one might hope that this would be self-correcting to some extent (people who do badly get dropped) it could still reduce the average accuracy of forecasts.