How to pick city policies, vs. how to pick the mayor. How to cook a meal, vs. how to pick a restaurant. How to win a game, vs. how to decide which team won. How to do a study, vs. how to pick a study to publish.
Markets doesn't really care about methods do they? How can a market communicate the method(s) being used to outsiders? It can't. The only real market output is a "price" that others can base future decisions on.
Great thing is that when you compare "markets" (which integrate information from whatever forces are bearing on it, statistical analysis might be one) with "decision makers" (people who sometimes literally make decisions based on their "gut" or a "roll of the dice" and where luck might win out for some time) the market should correct eventual "mistakes" quicker and be less wrong in general.
Funny how easily people can forget what markets really do!
One interesting aspect of this distinction, reflected I think in all four examples you gave, is that the relation between forum and method is not strictly hierarchical. Decisions as to how meals are cooked, for instance, mainly plays the role of background when choosing a restaurant, but in practice each particular choice of restaurant marginally alters the whole situation of how meals are cooked. Cf. discussions about law and constitution, Nomic, etc.
Along the same lines, while "we" might want to consider switching to prediction markets as the forum for evaluating new ideas, there is still the matter of who "we" is and figuring out that that's what we want. Of course a bunch of people could simply walk away from the academic forum model, but this represents a coordination problem (itself a reason to approach the transition initially through an academic lens just because it's the status quo).
In addition we may want to preserve an understanding of the relation between things we learn through prediction markets and things we have learned through the academic process. This could be dealt with (at least in principle) through either forum.
David, of course we should study and compare forums, ideally via direct horse-race comparisons of accuracy and resources used.
Simon, I agree that the degree to which the forecasting forum might induce non-forum actions is a relevant factor distinguishing forum environments. But there are a huge number of such factors. I disagree that a short post should explicitly mention all of them.
Robert, I agree that it is nice that academic forums encourage publication of methods, but I don't think that consideration is decisive.
I suggest prediction markets may be a better forum than academic journals for choosing forecasting methods
This glosses over a key distinction between prediction markets (or asset markets in general) and academic research. Academics publish their forecasting methods: here is how we collected and analyzed the data, etc. Speculators in markets would be crazy to do that. So you might get individual predictions from a market, but you won't get much insight into prediction methods.
I can't help but feel that there are certain areas where prediction markets will mobilize expertise, as you say, and others in which the prediction market itself will endogenously facilitate an outcome. Consistent with what Rajiv Sethi said recently. In which case you need to differentiate between those areas in which prediction markets could be interesting and helpful in predicting outcomes, and others in which they may also facilitate the realisation of certain outcomes.
That said your general point about the kind of forum being used is useful. What might be interesting, though, for the research Ayres talks about, is to assist us to understand the areas in which there are self-fulfilling predictions and areas where prediction markets actually mobilize expertise for useful predictions.
Robin, since you've made this argument in a blog post, aren't you implicitly suggesting that blogs should be the forum in we choose forums for forecasting?
Rather than criticize academics for their choice of medium for expression, let's focus on the substance of their arguments.
To continue your analogy, isn't it worthwhile to study whether good restaurant competition indeed induces good cooking? If we care a lot about good cooking, and hope to improve restaurant competition in order to improve the quality of cooking, then I expect restaurant competition would be a worthwhile object of study.
To pose a more formal question: should we use the same utility function for evaluating methods as we would for evaluating forums? That is, should we judge a forum by the methods it tends to choose? I think we should.
For example, if we have an idea of what is good cooking, then we should judge the state of the restaurant market by how its competitive process affects the quality of cooking. If your idea of good cooking includes the healthiness of the food, then you'll be dismayed about how restaurant competition in New York City pushes restaurants to make their food fattening.
Robin, I hope you'll agree that prediction markets are worth studying, and that some prediction markets are better than others. My own professional interest is in the microstructure of securities markets. The securities markets that prediction markets strive to emulate are rife with controversy, as they struggle to reconcile the competing interests of their various participants. After centuries of rule changes and decades of academic study, securities markets and their structure remain controversial and imperfect. Given their relative youth, we can expect prediction markets to be at least as imperfect, and at least as worthy of study and improvement.
I agree the world shouldn’t use a method just because academics say its great; rather, the world should try a method because it is persuaded by the arguments in its favor.
Markets doesn't really care about methods do they? How can a market communicate the method(s) being used to outsiders? It can't. The only real market output is a "price" that others can base future decisions on.
Great thing is that when you compare "markets" (which integrate information from whatever forces are bearing on it, statistical analysis might be one) with "decision makers" (people who sometimes literally make decisions based on their "gut" or a "roll of the dice" and where luck might win out for some time) the market should correct eventual "mistakes" quicker and be less wrong in general.
Funny how easily people can forget what markets really do!
One interesting aspect of this distinction, reflected I think in all four examples you gave, is that the relation between forum and method is not strictly hierarchical. Decisions as to how meals are cooked, for instance, mainly plays the role of background when choosing a restaurant, but in practice each particular choice of restaurant marginally alters the whole situation of how meals are cooked. Cf. discussions about law and constitution, Nomic, etc.
Along the same lines, while "we" might want to consider switching to prediction markets as the forum for evaluating new ideas, there is still the matter of who "we" is and figuring out that that's what we want. Of course a bunch of people could simply walk away from the academic forum model, but this represents a coordination problem (itself a reason to approach the transition initially through an academic lens just because it's the status quo).
In addition we may want to preserve an understanding of the relation between things we learn through prediction markets and things we have learned through the academic process. This could be dealt with (at least in principle) through either forum.
David, of course we should study and compare forums, ideally via direct horse-race comparisons of accuracy and resources used.
Simon, I agree that the degree to which the forecasting forum might induce non-forum actions is a relevant factor distinguishing forum environments. But there are a huge number of such factors. I disagree that a short post should explicitly mention all of them.
Robert, I agree that it is nice that academic forums encourage publication of methods, but I don't think that consideration is decisive.
I suggest prediction markets may be a better forum than academic journals for choosing forecasting methods
This glosses over a key distinction between prediction markets (or asset markets in general) and academic research. Academics publish their forecasting methods: here is how we collected and analyzed the data, etc. Speculators in markets would be crazy to do that. So you might get individual predictions from a market, but you won't get much insight into prediction methods.
Hi Robin,
I can't help but feel that there are certain areas where prediction markets will mobilize expertise, as you say, and others in which the prediction market itself will endogenously facilitate an outcome. Consistent with what Rajiv Sethi said recently. In which case you need to differentiate between those areas in which prediction markets could be interesting and helpful in predicting outcomes, and others in which they may also facilitate the realisation of certain outcomes.
That said your general point about the kind of forum being used is useful. What might be interesting, though, for the research Ayres talks about, is to assist us to understand the areas in which there are self-fulfilling predictions and areas where prediction markets actually mobilize expertise for useful predictions.
Robin, since you've made this argument in a blog post, aren't you implicitly suggesting that blogs should be the forum in we choose forums for forecasting?
Rather than criticize academics for their choice of medium for expression, let's focus on the substance of their arguments.
To continue your analogy, isn't it worthwhile to study whether good restaurant competition indeed induces good cooking? If we care a lot about good cooking, and hope to improve restaurant competition in order to improve the quality of cooking, then I expect restaurant competition would be a worthwhile object of study.
To pose a more formal question: should we use the same utility function for evaluating methods as we would for evaluating forums? That is, should we judge a forum by the methods it tends to choose? I think we should.
For example, if we have an idea of what is good cooking, then we should judge the state of the restaurant market by how its competitive process affects the quality of cooking. If your idea of good cooking includes the healthiness of the food, then you'll be dismayed about how restaurant competition in New York City pushes restaurants to make their food fattening.
Robin, I hope you'll agree that prediction markets are worth studying, and that some prediction markets are better than others. My own professional interest is in the microstructure of securities markets. The securities markets that prediction markets strive to emulate are rife with controversy, as they struggle to reconcile the competing interests of their various participants. After centuries of rule changes and decades of academic study, securities markets and their structure remain controversial and imperfect. Given their relative youth, we can expect prediction markets to be at least as imperfect, and at least as worthy of study and improvement.
I agree the world shouldn’t use a method just because academics say its great; rather, the world should try a method because it is persuaded by the arguments in its favor.