Does Profit Rate Insight Best?

Upcoming in Theory and Decision (in the same issue as I), David Johnstone asks a fundamental prediction market question: should we believe traders who make the most money, relative to traders with good statistical forecasting scores?  His answer is "No":

A gambler who wants to maximize future profits should trade on the advice of the analyst cum probability forecaster who records the best probability score, rather than the highest trading profits. 

The reason is simple: when a forecaster’s prediction falls within a bid-ask trading spread, he cannot improve his trading profit, but he can still improve his probability score.   The extreme case is even simpler:  if no market exists, he can make no trades, but he can still improve his score. 

Another reason to prefer a statistical score is that the trader’s goal might be something other than maximizing  the logarithm of their final trading profit.   Only for log utility traders with no bid-ask spread could profit reveal as much as a log scoring rule. 

On the other hand, if there is not just one statistical score to use, then a forecaster might have selected the score to show you that made them look best, reducing its info value.  This is much harder to do with trading profits. 

Finally, let me note that my automated market maker, which I call a "market scoring rule," has a zero bid ask spread.  So profit trading with it will rate someone just as well as a statistical score, at least if a combinatorial version is used that allows for all the combinations that a scoring system might allow. 

GD Star Rating
loading...
Tagged as:
Trackback URL:
  • Yan Li

    This might be too technical a subject for me to put my mind around it… How could a zero ask-bid spread yield a profit score equivalent to a statistical (or probability) score? Traders won’t trade when his/her profit forecast is negative…

  • http://profile.typekey.com/robinhanson/ Robin Hanson

    Yan, it is a technical question, so try following the link first and see if that explains it well enough.

  • Paul Gowder

    How are predictors revealing their predictions other than through their bets?

  • http://profile.typekey.com/robinhanson/ Robin Hanson

    Paul, predictors can just declare their predictions, which are then later collected and compared with the true answers, producing an accuracy score.

  • Paul Gowder

    Hmm… if there’s an active prediction market, why would (competent) predictors ever want to (honestly) declare their predictions? Or is the idea here that they’d work something like the people who publish stock picking advice now — that there’d be a layer of the prediction market that finds it more profitable to sell their opinions than to bet on them? (Does anyone know of any work comparing the records of the best of those public analysts to the best traders who don’t declare their picks?)

  • http://profile.typekey.com/robinhanson/ Robin Hanson

    Paul, there are several possible reasons to declare predictions, but this paper doesn’t care about the reason, it is looking at the consequences of such declarations.