The Trouble With Track Records

In December I argued:

I’d guess you can get 80% of the improvement that predict markets offer by using a much simpler solution: collect track records.  … When people make forecast-like-statements, write them down in a clear standardized form, and then check back later to see who was more accurate.   Along the way, create a consensus forecast by averaging recent forecasts, …  If you collect enough forecasts to evaluate accuracy, and reward accuracy well enough, people will try hard to be right, and you’ll learn what kinds of people to listen to.

Stock analysts are one of the few professions where we do keep track records.  But it appears that in the main clearinghouse for stock analyst picks, history has been edited to make favored analysts look better.   This illustrates an important advantages of betting markets over simple track records:  one side will complain loudly if the bet is edited to favor the other side.

Managers often accept that betting markets would give them more accurate organization forecasts, but complain that such markets are too complex, too disrupting of local culture, and leak info to outsiders.   So many are exploring various forms of "competitive forecasting," where people send their forecasts and updates to a black box than tells each person how well they are doing and whatever they need to know about the consensus.  This might work well if the people running the box can be trusted.   But I worry that black box bosses may have these biases:

  1. Choose and change the consensus measures to get the forecasts they want.
  2. Choose and change evaluation measures to make favored people look good.
  3. Let favored people better see the consensus, to make their forecasts better.
  4. Edit all the histories to make history appear they way they want.

It is much harder to bias real money betting markets in such ways, especially several independent markets connected via arbitrage. 

GD Star Rating
Tagged as:
Trackback URL:
  • There is a company ( which rates stock analysts on the accuracy of their predictions. I’d be curious to know if you think they have any or all of the four biases which you worry about above.

  • Rafe, if it was easy for observers to tell if these problems were happening, they wouldn’t be such problems. I have no special access to StarMine to be able to tell.

  • I’m puzzled as to why competing analysts don’t have enough incentive to complain about an alteration which causes a competitor to appear to have a better track record. Possibly because all the analysts who are aware that the history can be edited find it better to alter their own record than to blow the whistle on the others? It would seem unusual for a significant number of people to all act that way.
    For simple internal corporate predictions such as when will a project be completed, it still seems like prediction markets are more complex than needed. If companies adopt a simple rule of rewarding good forecasts with small bonuses, and allocate a fixed amount of total prediction-related bonuses (so that employees have a modest incentive to detect cheating), and each prediction is known to multiple employees, it seems like employees should be able to keep the system honest. Unless the rewards to inaccurate predictions are a good deal higher than my intuition leads me to expect.
    The black box proposal might reduce transparency enough to enable cheating. I’m unclear about how people try to justify a black box as opposed to keeping a record of predictions that is available to everyone on the project the predictions are about. Am I missing something in your black box reference?

  • Peter, managers are reluctant to lose control over information, and hence are reluctant to allow a public record of predictions.

  • Robin, what excuses do they give for hiding predictions from employees involved with the project in question? I’d expect that dealing with these excuses is no harder than dealing with the excuses for avoiding prediction markets.

  • Peter, managers usually hide lots of things from employees; they don’t need excuses to hide more, as the subject doesn’t come up.

  • For the purpose of predicting, if not selling forecasting methods to managers, transparency is desirable.. meaning that the inputs to each trader’s profit function, not to mention the function itself, should be known and verifiable by each trader. Whenever a trader’s p&l function, real or play, takes the predictions of others as an input, this transparency is lost.

    Auctions and algorithms can also be biased via large orders by “the house”.. but one must calibrate one’s cynicism and doubt.