Overcoming Bias

Share this post

The Trouble With Track Records

www.overcomingbias.com

Discover more from Overcoming Bias

This is a blog on why we believe and do what we do, why we pretend otherwise, how we might do better, and what our descendants might do, if they don't all die.
Over 11,000 subscribers
Continue reading
Sign in

The Trouble With Track Records

Robin Hanson
Mar 15, 2007
Share this post

The Trouble With Track Records

www.overcomingbias.com
7
Share

In December I argued:

I’d guess you can get 80% of the improvement that predict markets offer by using a much simpler solution: collect track records.  … When people make forecast-like-statements, write them down in a clear standardized form, and then check back later to see who was more accurate.   Along the way, create a consensus forecast by averaging recent forecasts, …  If you collect enough forecasts to evaluate accuracy, and reward accuracy well enough, people will try hard to be right, and you’ll learn what kinds of people to listen to.

Stock analysts are one of the few professions where we do keep track records.  But it appears that in the main clearinghouse for stock analyst picks, history has been edited to make favored analysts look better.   This illustrates an important advantages of betting markets over simple track records:  one side will complain loudly if the bet is edited to favor the other side.

Managers often accept that betting markets would give them more accurate organization forecasts, but complain that such markets are too complex, too disrupting of local culture, and leak info to outsiders.   So many are exploring various forms of "competitive forecasting," where people send their forecasts and updates to a black box than tells each person how well they are doing and whatever they need to know about the consensus.  This might work well if the people running the box can be trusted.   But I worry that black box bosses may have these biases:

  1. Choose and change the consensus measures to get the forecasts they want.

  2. Choose and change evaluation measures to make favored people look good.

  3. Let favored people better see the consensus, to make their forecasts better.

  4. Edit all the histories to make history appear they way they want.

It is much harder to bias real money betting markets in such ways, especially several independent markets connected via arbitrage. 

Share this post

The Trouble With Track Records

www.overcomingbias.com
7
Share
7 Comments
Share this discussion

The Trouble With Track Records

www.overcomingbias.com
Overcoming Bias Commenter
May 15

For the purpose of predicting, if not selling forecasting methods to managers, transparency is desirable.. meaning that the inputs to each trader's profit function, not to mention the function itself, should be known and verifiable by each trader. Whenever a trader's p&l function, real or play, takes the predictions of others as an input, this transparency is lost.

Auctions and algorithms can also be biased via large orders by "the house".. but one must calibrate one's cynicism and doubt.

Expand full comment
Reply
Share
Robin Hanson
May 15

Peter, managers usually hide lots of things from employees; they don't need excuses to hide more, as the subject doesn't come up.

Expand full comment
Reply
Share
5 more comments...
Top
New
Community

No posts

Ready for more?

© 2023 Robin Hanson
Privacy ∙ Terms ∙ Collection notice
Start WritingGet the app
Substack is the home for great writing