Does Profit Rate Insight Best?
Upcoming in Theory and Decision (in the same issue as I), David Johnstone asks a fundamental prediction market question: should we believe traders who make the most money, relative to traders with good statistical forecasting scores? His answer is "No":
A gambler who wants to maximize future profits should trade on the advice of the analyst cum probability forecaster who records the best probability score, rather than the highest trading profits.
The reason is simple: when a forecaster’s prediction falls within a bid-ask trading spread, he cannot improve his trading profit, but he can still improve his probability score. The extreme case is even simpler: if no market exists, he can make no trades, but he can still improve his score.
Another reason to prefer a statistical score is that the trader’s goal might be something other than maximizing the logarithm of their final trading profit. Only for log utility traders with no bid-ask spread could profit reveal as much as a log scoring rule.
On the other hand, if there is not just one statistical score to use, then a forecaster might have selected the score to show you that made them look best, reducing its info value. This is much harder to do with trading profits.
Finally, let me note that my automated market maker, which I call a "market scoring rule," has a zero bid ask spread. So profit trading with it will rate someone just as well as a statistical score, at least if a combinatorial version is used that allows for all the combinations that a scoring system might allow.
Paul, there are several possible reasons to declare predictions, but this paper doesn't care about the reason, it is looking at the consequences of such declarations.
Hmm... if there's an active prediction market, why would (competent) predictors ever want to (honestly) declare their predictions? Or is the idea here that they'd work something like the people who publish stock picking advice now -- that there'd be a layer of the prediction market that finds it more profitable to sell their opinions than to bet on them? (Does anyone know of any work comparing the records of the best of those public analysts to the best traders who don't declare their picks?)