Forum = Meta-Method

  1. How to pick city policies, vs. how to pick the mayor.
  2. How to cook a meal, vs. how to pick a restaurant.
  3. How to win a game, vs. how to decide which team won.
  4. How to do a study, vs. how to pick a study to publish.

These are four examples of methods vs. forums.  Methods are ways to do things; forums are ways to pick who decides what to do.  Yes, in a sense forums are methods, since choosing who decides indirectly picks what to do.  But that is what makes forums powerful; good forums induce people to find good methods.  Good  elections induces good city policies, good restaurant competition induces good cooking, good game rules induce good play, and good journal review induces good articles.

To me, prediction markets are mostly interesting as forums, not methods.  Alas many seem to confuse the two.  E.g., Ian Ayres at Freakonomics:

One of the great unresolved questions of predictive analytics is trying to figure out when prediction markets will produce better predictions than good old-fashion mining of historic data. … We are about to have a test of these two competing approaches … a cool Supreme Court fantasy league, where anybody can make predictions about how Supreme Court justices will vote on particular cases. …

[Will aggregate] predictions of the league [be] more accurate than the predictions of a statistical algorithm developed by [five stat experts?] … The fantasy league predictions would probably be more accurate if market participants had to actually put their money behind their predictions. … Statistical predictions could probably be improved if they relied on more recent data and controlled for more variables.

More meta-methodological comparisons like these … will also shed light on whether market participants will learn to efficiently incorporate the results of statistical prediction into their own assessments. At the moment, individual decision-makers tend to improve their prediction when given statistical aids; but they still tend to wave off the statistical prediction too often.

James Surowiecki’s book seems responsible for so many folks equating “prediction markets” with “wisdom of crowd” averages of non-expert more-intuitive opinion, vs. formal expert analysis.  Averaging popular opinion may be an interesting method, as is statistical analysis, but comparing these does not evaluate prediction markets as forums.

“Prediction markets” started from speculative markets, e.g. stocks, where accuracy comes much less from non-expert participation and much more from participants with incentives to self-select as experts.  Any team that considers itself expert enough can pay to prove itself, but in fact most teams stay away and prices tend to be dominated by real experts, who get paid and really know better than most.

Prediction markets aren’t about emphasizing ordinary Joes over credentialed bigshots; they are about emphasizing whomever tends to be right.  Simple opinion averages maybe be reasonable indicators of crowd wisdom, but they have too little of the forum-ness of letting self-selected expert teams come to dominate.

It seems to me that when academics like Aryes call for academic studies of prediction markets as methods, instead of as forums, they are implicitly suggesting that current academic institutions should be the forum in we choose forecasting methods.  If academic journals prefer a method, they suggest, that’s the method the world should use.

In contrast, I suggest prediction markets may be a better forum than academic journals for choosing forecasting methods.  Maybe the world shouldn’t use a method just because academics say its great; maybe those impressed with a method should have to put their money where their mouth is and trade on that method’s forecasts in prediction markets.  Maybe the rest of us should just accept prediction market prices as our best estimates; if and when prediction market prices become dominated by traders using a method, that is when the rest of us will have implicitly accepted that method as best.

GD Star Rating
Tagged as: ,
Trackback URL:
  • Pingback: Midas Oracle .ORG()

  • David J

    Robin, since you’ve made this argument in a blog post, aren’t you implicitly suggesting that blogs should be the forum in we choose forums for forecasting?

    Rather than criticize academics for their choice of medium for expression, let’s focus on the substance of their arguments.

    To continue your analogy, isn’t it worthwhile to study whether good restaurant competition indeed induces good cooking? If we care a lot about good cooking, and hope to improve restaurant competition in order to improve the quality of cooking, then I expect restaurant competition would be a worthwhile object of study.

    To pose a more formal question: should we use the same utility function for evaluating methods as we would for evaluating forums? That is, should we judge a forum by the methods it tends to choose? I think we should.

    For example, if we have an idea of what is good cooking, then we should judge the state of the restaurant market by how its competitive process affects the quality of cooking. If your idea of good cooking includes the healthiness of the food, then you’ll be dismayed about how restaurant competition in New York City pushes restaurants to make their food fattening.

    Robin, I hope you’ll agree that prediction markets are worth studying, and that some prediction markets are better than others. My own professional interest is in the microstructure of securities markets. The securities markets that prediction markets strive to emulate are rife with controversy, as they struggle to reconcile the competing interests of their various participants. After centuries of rule changes and decades of academic study, securities markets and their structure remain controversial and imperfect. Given their relative youth, we can expect prediction markets to be at least as imperfect, and at least as worthy of study and improvement.

    I agree the world shouldn’t use a method just because academics say its great; rather, the world should try a method because it is persuaded by the arguments in its favor.

  • Simon

    Hi Robin,

    I can’t help but feel that there are certain areas where prediction markets will mobilize expertise, as you say, and others in which the prediction market itself will endogenously facilitate an outcome. Consistent with what Rajiv Sethi said recently. In which case you need to differentiate between those areas in which prediction markets could be interesting and helpful in predicting outcomes, and others in which they may also facilitate the realisation of certain outcomes.

    That said your general point about the kind of forum being used is useful. What might be interesting, though, for the research Ayres talks about, is to assist us to understand the areas in which there are self-fulfilling predictions and areas where prediction markets actually mobilize expertise for useful predictions.

  • I suggest prediction markets may be a better forum than academic journals for choosing forecasting methods

    This glosses over a key distinction between prediction markets (or asset markets in general) and academic research. Academics publish their forecasting methods: here is how we collected and analyzed the data, etc. Speculators in markets would be crazy to do that. So you might get individual predictions from a market, but you won’t get much insight into prediction methods.

  • David, of course we should study and compare forums, ideally via direct horse-race comparisons of accuracy and resources used.

    Simon, I agree that the degree to which the forecasting forum might induce non-forum actions is a relevant factor distinguishing forum environments. But there are a huge number of such factors. I disagree that a short post should explicitly mention all of them.

    Robert, I agree that it is nice that academic forums encourage publication of methods, but I don’t think that consideration is decisive.

  • Benquo

    One interesting aspect of this distinction, reflected I think in all four examples you gave, is that the relation between forum and method is not strictly hierarchical. Decisions as to how meals are cooked, for instance, mainly plays the role of background when choosing a restaurant, but in practice each particular choice of restaurant marginally alters the whole situation of how meals are cooked. Cf. discussions about law and constitution, Nomic, etc.

    Along the same lines, while “we” might want to consider switching to prediction markets as the forum for evaluating new ideas, there is still the matter of who “we” is and figuring out that that’s what we want. Of course a bunch of people could simply walk away from the academic forum model, but this represents a coordination problem (itself a reason to approach the transition initially through an academic lens just because it’s the status quo).

    In addition we may want to preserve an understanding of the relation between things we learn through prediction markets and things we have learned through the academic process. This could be dealt with (at least in principle) through either forum.

  • ERIC

    Markets doesn’t really care about methods do they? How can a market communicate the method(s) being used to outsiders? It can’t. The only real market output is a “price” that others can base future decisions on.

    Great thing is that when you compare “markets” (which integrate information from whatever forces are bearing on it, statistical analysis might be one) with “decision makers” (people who sometimes literally make decisions based on their “gut” or a “roll of the dice” and where luck might win out for some time) the market should correct eventual “mistakes” quicker and be less wrong in general.

    Funny how easily people can forget what markets really do!

  • Pingback: Robin Hanson on Prediction Markets as Decision Tools | Prediction Markets Blog()

  • Pingback: Recomendaciones « intelib()

  • Pingback: Overcoming Bias : New Scientist Contest()