Overcoming Bias

Share this post

Does Profit Rate Insight Best?

www.overcomingbias.com

Does Profit Rate Insight Best?

Robin Hanson
Dec 3, 2006
Share this post

Does Profit Rate Insight Best?

www.overcomingbias.com

Upcoming in Theory and Decision (in the same issue as I), David Johnstone asks a fundamental prediction market question: should we believe traders who make the most money, relative to traders with good statistical forecasting scores?  His answer is "No":

A gambler who wants to maximize future profits should trade on the advice of the analyst cum probability forecaster who records the best probability score, rather than the highest trading profits. 

The reason is simple: when a forecaster’s prediction falls within a bid-ask trading spread, he cannot improve his trading profit, but he can still improve his probability score.   The extreme case is even simpler:  if no market exists, he can make no trades, but he can still improve his score. 

Another reason to prefer a statistical score is that the trader’s goal might be something other than maximizing  the logarithm of their final trading profit.   Only for log utility traders with no bid-ask spread could profit reveal as much as a log scoring rule. 

On the other hand, if there is not just one statistical score to use, then a forecaster might have selected the score to show you that made them look best, reducing its info value.  This is much harder to do with trading profits. 

Finally, let me note that my automated market maker, which I call a "market scoring rule," has a zero bid ask spread.  So profit trading with it will rate someone just as well as a statistical score, at least if a combinatorial version is used that allows for all the combinations that a scoring system might allow. 

Share this post

Does Profit Rate Insight Best?

www.overcomingbias.com
Comments
TopNewCommunity

No posts

Ready for more?

© 2023 Robin Hanson
Privacy ∙ Terms ∙ Collection notice
Start WritingGet the app
Substack is the home for great writing