Discover more from Overcoming Bias
How to pick city policies, vs. how to pick the mayor.
How to cook a meal, vs. how to pick a restaurant.
How to win a game, vs. how to decide which team won.
How to do a study, vs. how to pick a study to publish.
These are four examples of methods vs. forums. Methods are ways to do things; forums are ways to pick who decides what to do. Yes, in a sense forums are methods, since choosing who decides indirectly picks what to do. But that is what makes forums powerful; good forums induce people to find good methods. Good elections induces good city policies, good restaurant competition induces good cooking, good game rules induce good play, and good journal review induces good articles.
To me, prediction markets are mostly interesting as forums, not methods. Alas many seem to confuse the two. E.g., Ian Ayres at Freakonomics:
One of the great unresolved questions of predictive analytics is trying to figure out when prediction markets will produce better predictions than good old-fashion mining of historic data. … We are about to have a test of these two competing approaches … a cool Supreme Court fantasy league, where anybody can make predictions about how Supreme Court justices will vote on particular cases. …
[Will aggregate] predictions of the league [be] more accurate than the predictions of a statistical algorithm developed by [five stat experts?] … The fantasy league predictions would probably be more accurate if market participants had to actually put their money behind their predictions. … Statistical predictions could probably be improved if they relied on more recent data and controlled for more variables.
More meta-methodological comparisons like these … will also shed light on whether market participants will learn to efficiently incorporate the results of statistical prediction into their own assessments. At the moment, individual decision-makers tend to improve their prediction when given statistical aids; but they still tend to wave off the statistical prediction too often.
James Surowiecki’s book seems responsible for so many folks equating “prediction markets” with “wisdom of crowd” averages of non-expert more-intuitive opinion, vs. formal expert analysis. Averaging popular opinion may be an interesting method, as is statistical analysis, but comparing these does not evaluate prediction markets as forums.
“Prediction markets” started from speculative markets, e.g. stocks, where accuracy comes much less from non-expert participation and much more from participants with incentives to self-select as experts. Any team that considers itself expert enough can pay to prove itself, but in fact most teams stay away and prices tend to be dominated by real experts, who get paid and really know better than most.
Prediction markets aren’t about emphasizing ordinary Joes over credentialed bigshots; they are about emphasizing whomever tends to be right. Simple opinion averages maybe be reasonable indicators of crowd wisdom, but they have too little of the forum-ness of letting self-selected expert teams come to dominate.
It seems to me that when academics like Aryes call for academic studies of prediction markets as methods, instead of as forums, they are implicitly suggesting that current academic institutions should be the forum in we choose forecasting methods. If academic journals prefer a method, they suggest, that’s the method the world should use.
In contrast, I suggest prediction markets may be a better forum than academic journals for choosing forecasting methods. Maybe the world shouldn’t use a method just because academics say its great; maybe those impressed with a method should have to put their money where their mouth is and trade on that method’s forecasts in prediction markets. Maybe the rest of us should just accept prediction market prices as our best estimates; if and when prediction market prices become dominated by traders using a method, that is when the rest of us will have implicitly accepted that method as best.
Forum = Meta-Method
Markets doesn't really care about methods do they? How can a market communicate the method(s) being used to outsiders? It can't. The only real market output is a "price" that others can base future decisions on.
Great thing is that when you compare "markets" (which integrate information from whatever forces are bearing on it, statistical analysis might be one) with "decision makers" (people who sometimes literally make decisions based on their "gut" or a "roll of the dice" and where luck might win out for some time) the market should correct eventual "mistakes" quicker and be less wrong in general.
Funny how easily people can forget what markets really do!
One interesting aspect of this distinction, reflected I think in all four examples you gave, is that the relation between forum and method is not strictly hierarchical. Decisions as to how meals are cooked, for instance, mainly plays the role of background when choosing a restaurant, but in practice each particular choice of restaurant marginally alters the whole situation of how meals are cooked. Cf. discussions about law and constitution, Nomic, etc.
Along the same lines, while "we" might want to consider switching to prediction markets as the forum for evaluating new ideas, there is still the matter of who "we" is and figuring out that that's what we want. Of course a bunch of people could simply walk away from the academic forum model, but this represents a coordination problem (itself a reason to approach the transition initially through an academic lens just because it's the status quo).
In addition we may want to preserve an understanding of the relation between things we learn through prediction markets and things we have learned through the academic process. This could be dealt with (at least in principle) through either forum.