Reply to Moldbug

A Mencius Moldbug has written a confused and rambling 7400 word critique of futarchy. But since Mencius seems to have passion and potential, let me try to communicate.  Most readers may prefer to skip this post; it will get tedious.

Prediction markets are a fine idea, whereas decision markets are… well… retarded. … Almost every conceivable application of a decision market … does not produce accurate predictions.

So if PM good, DM bad, your complaints should focus on features that distinguish decision and prediction markets, right?

For a market to produce accurate predictions, there must be genuine experts in the market, and they must be substantially better-funded than the morons.

OK, except that morons may largely cancel each other, in which case you don't need as many non-morons.  But this issue applies equally to prediction and decision markets, doesn't it?

If no one has ever seen the Emperor of China's nose, can a prediction market predict its length? … The worst case is that in which nobody has any way of actually calculating the prediction, but no one in the market is sure that this is the case. Your market signal will look exactly like that of an accurate prediction market, but predict nothing at all.

You can ask for a full probability distribution.  If speculators know they don't know anything, then they will give you a broad distribution that expresses a lot of uncertainty.  This is them telling you they don't know much.  In your nose example, they may just give you the distribution over nose sizes for elite Chinese.  And how is this issue different for prediction vs. decision markets?

In an old and healthy prediction market we would indeed expect to see accurate predictions. But only in the case in which Darwinian forces have been operating for a substantial period of time.

Trader experience does seem to improve their collective accuracy.  But this hardly means accuracy was zero initially.  By now we have a great deal of experience in field and lab prediction markets where accuracy was valuable from day one.  And how does this issue distinguish prediction vs. decision markets?

 The adaptive explanation of market efficiency tells us why prediction markets with pretend money are a complete waste of time. These just aggregate the whims of the participants.

Play money markets do often fail badly, but many of them have done very well.  All else equal I prefer real money markets, but "complete waste of time" is way too strong.  And how does this distinguish prediction vs. decision markets?

A prediction market only makes valid predictions if it can resist feedback, where feedback is any effect of the market's result on the interests of the players. … With feedback, players with a financial interest in the effect of the prediction have an interest which conflicts with the efficiency of the market.

There are several possible concerns you might have here.  First, traders might sabotage events in the real world to improve their forecast accuracy.  This can be a problem for any forecasting mechanism that rewards accurate contributions, and there are several ways to deal with it.  Also, traders might try to distort the market price because that price influences decisions.  Theory, lab data, and field data suggest this is not much of a problem.  And how do these issues distinguish prediction vs. decision markets?

A prediction market, like any other market, functions only in the general absence of asymmetrical information. It is with some pain that I absorb the realization that a member of the George Mason School is unable to correctly apply this concept. … The rational approach to a market in which other players have more information than you is not to play. … This is one of the many reasons why insider trading is illegal.

Er, in pretty much every speculative market that has ever existed, traders have had differing relevant info.  Identical info is a very rare situation.  Some traders had more total info, and the traders who had less would have been well advised to leave.  And how does this distinguish prediction vs. decision markets?

Futarchy is… retarded, … because … It must be trained. During the training period, it is an inaccurate prediction market. … But who wants to run an inaccurate futarchy for a training period? … Testing the accuracy of a prediction market … is close to impossible.

After the fact it is quite easy to test for forecast accuracy.  And even if initial accuracy was poor, why not begin if we expect accuracy to greatly improve?  And how does this distinguish prediction vs. decision markets?

In order to make an accurate bet that some policy or other will increase GDP, as opposed to no policy at all, you first need an accurate estimate of GDP assuming no policy change at all. Whose error bars are shorter than the impact you expect from the policy change.  In other words, Professor Hanson is basically expecting the AM antenna on his ghetto blaster to pick up signals from Alpha Centauri.

We can sensibly compare small differences in conditional expectations even when conditional variances are high.  For example, even when you cannot predict the date of your death well, you can reasonable expect a longer lifespan if you exercise today than if you do not exercise today.  The difference in these two expectations can be tiny compared to the variance in your realized lifespan, and yet you can have reasonable beliefs about this difference, and act on them.  See also the presidential nomination decision markets which have been run for many years by IEM and Intrade as proof that decision markets are mechanically feasible, even in situations of great uncertainty.

Professors, Hanson and Abramowicz. Their primary angle seems to be in recommending decision markets to USG.

There is now a prediction market industry, where I do a lot of consulting, most of which is to private companies.  Most of this consulting is on prediction markets, but some are decision markets. 

Consider Professor Hanson's favorite statistic – GDP. If we translate GDP into Apple terms, it is gross revenue. GDP is simply the total sales of all American companies. So suppose our man, at Apple, had access to a perfect Hansonian decision market, which for any policy could calculate whether or not that policy would increase Apple's sales. … Many bad decisions increase gross revenue. What about costs? What about quality? What about customer service and brand reputation? What about employee morale? Etc, etc, etc.

For firms I recommend forecasting profits or stock price, not revenue.  For nations I suggest a full national welfare; GDP is only a first step to such a measure.

GD Star Rating
a WordPress rating system
Tagged as:
Trackback URL:
  • josh

    Concise.

    I wonder if you would agree with MM that crazy ideas like yours don’t face any genuine intellectual hostility within academia.

  • http://billmill.org Bill Mill

    I second Josh’s question.

  • jimrandomh

    Robin Hanson, you have fallen into an affective death spiral around prediction markets. Somewhere you went from believing that prediction markets work for certain things, with certain types of participants, under certain circumstances, to believing that prediction markets always work.

    The problem with prediction markets is that they create a financial incentive to spread disinformation. Consider this interview,
    http://www.thedailyshow.com/video/index.jhtml?videoId=221517&title=jim-cramer-unedited-interview
    Starting at about 7:50, Jim Cramer, who was a hedge fund trader, admitted to successfully spreading false rumors about Apple in order to capitalize on changes in its stock price. Earlier in that interview, he says that sort of thing is banned, but that the ban is impossible to enforce. If predictions behave like the stock market did, then the net effect of creating a prediction market will be to create one piece of good information, at the expense of promoting disinformation everywhere else. I do not consider that an acceptable trade-off.

  • http://macroethics.blogspot.com nazgulnarsil

    on reading the title I had greatly hoped that the two of you had gotten into a dialogue over something much more meaty than decision markets.

  • http://timtyler.org/ Tim Tyler

    If you play in the stock markets you can expect returns in line with inflation. Prediction and decision markets are zero-sum games though – and thus seem likely to be fundamentally less attractive. There, the people most likely to want to play are those with a vested interest in the outcome.

  • http://profile.typepad.com/robinhanson Robin Hanson

    Josh, that sounds like too strong a claim for me to endorse.

    Jim, of course prediction markets don’t always work. Often existing institutions do a good job collecting and summarizing info into consensus estimates, and in such cases prediction markets just tell you the same thing at some added cost. If you want to know what I had for breakfast, just ask me; no need for a prediction market. One example of someone trying to spread false rumors hardly proves that on average prediction markets make estimates less accurate.

    Tim, betting markets are no more zero sum than stock markets. You can bet a stock index fund, and then the bets grow just as fast as stocks on average. And patrons can subsidize betting markets.

  • Arthur B.

    Moldbug is confused about insider trading. Insider trading isn’t illegal because the traders would be unfairly more informed than the other market participants. It’s illegal because the shareholders who put the management of the company in place require that the management disclose any information they have to them for free before they trade on it.

    http://www.vnunet.com/vnunet/news/2209899/hacker-keep-profits

    A Ukrainian hacker stole information from a company and traded on it. The only crime he committed was breaching in a computer system, NOT insider trading although he obviously had access to privileged information.

    Generally though, I am also skeptical of futarchy. For example, I am not convinced that one can devise good tests for policies. If the goals are too specific metrics, horrible thing will happen (many policies guarantee low unemployment, most of them are BAD). If they are too general (say people report they are happy), the effect of a specific policy may be too diluted by other policies to allow a reliable payoff.

    I also find the idea of voting on values a bit repulsive.

  • http://unqualified-reservations.blogspot.com Moldbug

    OK, except that morons may largely cancel each other, in which case you don’t need as many non-morons. But this issue applies equally to prediction and decision markets, doesn’t it?

    “May”? This issue does not apply equally to prediction and decision markets. Because prediction markets are not making decisions. Unless you’d like to defend “may” in the context of controlling government decisions.

    If a long attention span is not your strong point, I sympathize. This is the entire problem with your proposal. Even if you accept the need for mechanical decision formulation, a new mechanical decision procedure will not be accepted unless its reliability rests on something more than “may.”

    Trader experience does seem to improve their collective accuracy. But this hardly means accuracy was zero initially.

    No. And just because the engine is attached to your 747 with duct tape, doesn’t mean it will fall off.

    First, traders might sabotage events in the real world to improve their forecast accuracy. This can be a problem for any forecasting mechanism that rewards accurate contributions, and there are several ways to deal with it.

    This was not a complaint I raised, but I can only applaud you for adding it.

    Also, traders might try to distort the market price because that price influences decisions. Theory, lab data, and field data suggest this is not much of a problem.

    And cold weather has never been a problem with our O-rings. I’m an engineer. Inductive data doesn’t impress me.

    Unless you object, I’ll take your lack of response to my deductive analysis of this case as agreement.

    And how do these issues distinguish prediction vs. decision markets?

    See under “may.”

    Er, in pretty much every speculative market that has ever existed, traders have had differing relevant info.

    Not systematically and predictably. Sure, people break the law. But that doesn’t mean there is no law. It also doesn’t mean that however well these markets work, they wouldn’t work better if no one was breaking the law.

    After the fact it is quite easy to test for forecast accuracy. And even if initial accuracy was poor, why not begin if we expect accuracy to greatly improve? And how does this distinguish prediction vs. decision markets?

    See under “may.”

    We can sensibly compare small differences in conditional expectations even when conditional variances are high. For example, even when you cannot predict the date of your death well, you can reasonable expect a longer lifespan if you exercise today than if you do not exercise today. The difference in these two expectations can be tiny compared to the variance in your realized lifespan, and yet you can have reasonable beliefs about this difference, and act on them.

    The signal may exist. The question, however, is whether the market can receive it. When the station is broadcasting from Alpha Centauri, I doubt it.

    You can have these reasonable beliefs. Great! Have them. But for the Darwinian iteration of a market to select in favor of those who have reasonable beliefs, and against those who have unreasonable beliefs, those who have reasonable beliefs must profit selectively over those who have unreasonable beliefs. If the effect is small, the selective pressure is small. Which means your antenna won’t pick up Alpha Centauri.

    And if this adaptive iteration has not happened, all your “market” is doing is taking a vote. You’re right back to democracy.

    See also the presidential nomination decision markets which have been run for many years by IEM and Intrade as proof that decision markets are mechanically feasible.

    Now you’re the one confusing prediction and decision markets! Do we even mean the same things by these words? To me, a decision market is a prediction market which is used to make decisions, ie, a

    There is now a prediction market industry, where I do a lot of consulting, most of which is to private companies.

    Wonderful! I have no general case against prediction markets, although as I say they are not magic. For many purposes, they are excellent. I am sure you have much highly valuable advice to provide on the subject. And, I mean, you’d have to work pretty hard to be as retarded as, say, McKinsey:

    http://www.theatlantic.com/doc/200606/stewart-business/

  • http://unqualified-reservations.blogspot.com Moldbug

    Oh, and I missed this:

    You can ask for a full probability distribution. If speculators know they don’t know anything, then they will give you a broad distribution that expresses a lot of uncertainty. This is them telling you they don’t know much. In your nose example, they may just give you the distribution over nose sizes for elite Chinese.

    This suggests to me that it’s been a while since you read Feynman. My fault – I should have been clearer. The entire point of Feynman’s story is that no one knows anything about the Emperor of China’s nose. It’s a gedankenexperiment, not a problem in physical anthropology.

    My worst-case scenario was a market in which no one knows anything, but no one knows that no one knows anything. This is a pure noise market. Moreover, since no one knows it is a pure noise market, no one will design it to generate a full probability distribution.

  • http://profile.typepad.com/robinhanson Robin Hanson

    Moldbug, Morons and non-morons, which I often call sheep and wolves, are not distributed independently across topics. The wolves are consistently attracted to the sheep, and the net effect of having more wolves and more sheep is to have more accurate markets. Having a decision depend on a market induces people who want to manipulate the price, who are in effect sheep, since they attract wolves. You are somehow worried that decision markets that determine government policy would attract sheep without wolves, but I can’t imagine why.

    I’ve been trying here, but I do have limited patience; it is your job to make a clear pointed objection, not my job to try to discern it in all your varied comments. You think prediction markets are fine, but decision markets are not, all because of some “deductive analysis” I can’t find. You are not interested in any of our lab or field data or the standard theory economists use to make sense of this data, in either the prediction or decision case. So I’m stumped. Explain yourself clearly or shut up.

  • http://unqualified-reservations.blogspot.com Moldbug

    The cut-off sentence: “a decision market is a prediction market which is used to make decisions, ie, through a mechanical linkage as in futarchy.”

    And I also missed

    For nations I suggest a full national welfare; GDP is only a first step to such a measure.

    You would, wouldn’t you? Have you considered the King of Bhutan’s little innovation, “Gross National Happiness?” I’ll bet you’re already a fan:

    http://www.nytimes.com/2009/05/07/world/asia/07bhutan.html

    I’m quite confident that you have no idea at all of why anyone would consider the idea of constructing a national goodness index, and governing to maximize it, in any way, shape, or form bizarre, Orwellian, or just plain… retarded. And that’s okay. Perhaps that’s how it should stay.

    If not, I suspect you also skimmed the Carlyle excerpt at the end of my post. If you reached it. If you tire of responding to me, for which I can certainly excuse you in advance, I’m sure your readers would enjoy a response to Carlyle. Chapter 2, “Statistics,” in the pamphlet on Chartism.

  • http://unqualified-reservations.blogspot.com Moldbug

    It is your job to make a clear pointed objection, not my job to try to discern it in all your varied comments.

    I’ve responded clearly to all your points – and you know it.

  • http://unqualified-reservations.blogspot.com Moldbug

    Morons and non-morons, which I often call sheep and wolves, are not distributed independently across topics. The wolves are consistently attracted to the sheep, and the net effect of having more wolves and more sheep is to have more accurate markets. Having a decision depend on a market induces people who want to manipulate the price, who are in effect sheep, since they attract wolves. You are somehow worried that decision markets that determine government policy would attract sheep without wolves, but I can’t imagine why.

    In plain English: the initial state of any market is a mix of accurate and inaccurate predictors. The accuracy of the market depends on the proportion of sheep to wolves, which the designer of the market does not control. Therefore, the designer cannot promise accurate initial results.

    In a futarchy, which is a market in which government decisions are mechanically linked to the results of a prediction market, accurate initial results are essential, because the random initial mix of sheep and wolves will guide initial decisions. I can’t imagine how this could be acceptable.

    Moreover, a futarchy will contain a third class of animal: bears, animals that are willing to lose money by making intentionally bad bets because they profit from bad policies. Your explanation of why the results of a futarchy will not be dominated by bears is that the bears will lose too much money to the wolves, and be unable to continue. Over time, this may be true, but it is unlikely to be true in the initial state.

    Thus, prediction markets differ from futarchy in that the operator of a prediction market can afford to take its results with a grain of salt. As I’m sure your customers do.

    Moreover, you seem to be using an oversimplified model which assumes, unjustifiably, that the errors made by sheep follow a normal distribution – whose center is the just price as defined by the wolves. I hope I am reading you wrong here, but if not I urge you to abandon this assumption, which is obviously unjustified by reality.

  • http://profile.typepad.com/equimarginal Zac Gochenour

    Moldbug, about which topic does no one know anything?

    You do realize you are making no sense, right?

    I wish you did, since I like to read smart debate about topics I am interested in. I agree with you that there doesn’t seem to be enough genuine intellectual hostility. There’s plenty of unintellectual hostility to go around, though.

  • http://unqualified-reservations.blogspot.com Moldbug

    Let me put this discussion slightly in context by explaining the difference between a trained market, one that has undergone adaptive selection, and an untrained market, eg in the initial state.

    Trained markets (such as Wall Street or Vegas) achieve remarkable results because, over generational periods of time (think of how long people have been betting on stocks, or on the NBA) they foster a remarkable quantity of expertise. They breed super-wolves, who feast on the sheep. The market is not magic. It does not create this information. It merely aggregates it. It is the humans who are the experts.

    Untrained markets aggregate the opinions of their participants, just like trained ones. The aggregation step is not what makes the market expert – the expertise is. A market mechanism works perfectly well as a way of voting on probabilities, but voting is all it is. Both trained and untrained markets are useful – but they are very different things.

    It is all too easy to confuse these very different things, and associate the remarkable expertise that collects in a trained market with the simple betting mechanism shared by both. Once a logical mind has made this mistake, it is very easy to come up with ideas such as futarchy.

  • http://unqualified-reservations.blogspot.com Moldbug

    Moldbug, about which topic does no one know anything?

    Why, planetary gormball, of course. But I thought you’d read my post? Those of us who like to think, Zac, like to engage in what we call “thought-experiments.” Here’s another one for you.

    Consider the classic MBA experiment, in which the professor shows the class a jar full of jellybeans and asks them to guess how many are in the jar. At least according to folklore – I don’t have an MBA – the class’s answers always cluster in a normal distribution around the right answer. This fact, which must arise from some actual human expertise or other (probably a volume-calculation “module” in the brain), is often cited as folk wisdom that markets can produce good information, even though no one in the market has any – garbage in, filet-mignon out.

    Now, consider an alternate version of the experiment, in which the professor says: “Class, I have a jar of jellybeans under my desk. How many jellybeans are in it?” That’s a market in which no one knows anything. Don’t expect an accurate aggregate.

    Are there any other points on which I’m making no sense, Zac? Do tell.

  • GreedyAlgorithm

    >> Now, consider an alternate version of the experiment, in which the professor says: “Class, I have a jar of jellybeans under my desk. How many jellybeans are in it?” That’s a market in which no one knows anything. Don’t expect an accurate aggregate.

    This is an experiment we can do. Would you like to refine it before betting against an accurate aggregate? Hint: you would.

  • Jeff

    Robin, I really appreciate you having this conversation. Here’s my problem:

    “Having a decision depend on a market induces people who want to manipulate the price, who are in effect sheep, since they attract wolves. You are somehow worried that decision markets that determine government policy would attract sheep without wolves, but I can’t imagine why.”

    You’re right that decision markets would attract sheep and wolves. The question I have is why you’d imagine that the benefit to the wolves would outweigh the downside to the sheep. Under decision markets, if I have enough money, it seems that I can purchase government actions that gain me much more money — the wolves are satiated by my price, but I don’t have to stop being involved (so feedback for losers is broken).

    The example, of course, is a large corporation or extremely wealthy individual who spends a fraction of their wealth winning in a decision market to push through a policy that dramatically increases only their wealth. As Half Sigma would say, value transference under the control of the top managers, and you’ve transferred top management of the country to those with money.

    You may well believe that there will be enough wolves who will see the benefit to betting against any such proposition to effectively prevent the proposition from happening. I cannot imagine how you could have the evidence to support such a belief, considering the consequences of being wrong. Futarchy, like many other proposed governmental systems, seems to fail the Bruce Schneier test: I haven’t seen evidence that you’ve thought about it the way an attacker would think about it. Please show us why you believe it’s more resistant to gaming than the current system?

  • http://intellectual-detox.com Devin Finbarr

    Moldbug-

    In a futarchy, which is a market in which government decisions are mechanically linked to the results of a prediction market, accurate initial results are essential, because the random initial mix of sheep and wolves will guide initial decisions. I can’t imagine how this could be acceptable.

    You could start the decision market as a prediction market. At first people would bet on whether policy A will have results X. Once the wolves have eaten the sheep, and the market has learned to make very accurate predictions, you can then hook the market up to the machine and have the bets actually dictate the policy.

    Your other points are spot on though. There are very few decisions that can be defined as a multiple choice problem with a well defined result. And if you could trust the leaders to phrase the questions, the choices, and the result measurements, you could trust them to use the prediction markets as a tool, rather than a machine.

  • William

    “That’s a market in which no one knows anything.”

    I know the jar of jellybeans is small enough to fit in a desk. I know how big jars usually are.

  • http://macroethics.blogspot.com nazgulnarsil

    here’s a question…

    does the informed bettor make a bet on what he actually believes the correct answer to be? or does he place a bet to make the aggregate more accurate?

    example: I believe the correct answer to be 0, someone else has bet -1. Do I bet 0 or 1?

    of course this hinges on where the bettor is making his profit. Does he profit from a correct outcome or does he profit from being correct regardless of outcome?

  • http://dogofjustice.livejournal.com Dog of Justice

    You may well believe that there will be enough wolves who will see the benefit to betting against any such proposition to effectively prevent the proposition from happening. I cannot imagine how you could have the evidence to support such a belief, considering the consequences of being wrong. Futarchy, like many other proposed governmental systems, seems to fail the Bruce Schneier test: I haven’t seen evidence that you’ve thought about it the way an attacker would think about it. Please show us why you believe it’s more resistant to gaming than the current system?

    Well said, Jeff; this is my objection to futarchy as well. As far as I can tell, the results of prediction markets can only be mechanically applied to relatively small-scale problems. If it’s ever known in advance that a politically important decision hinges on a market price, “bears” should be expected… and I would expect the strongest group of them to win.

  • Cyan

    does the informed bettor make a bet on what he actually believes the correct answer to be? or does he place a bet to make the aggregate more accurate?

    The informed bettor makes a bet when her personal odds for the outcome differ from the market-implied odds. For example, if her personal odds are 4 to 1 against and the market is offering 8 to 1 against, she will take the proffered bet. Note that she’s betting against what she believes the final outcome will most probably be. Supposing she is accurate, over the long run she will lose four such bets for every one she wins, but when she wins she’ll make back twice the money she lost.

  • http://entitledtoanopinion.wordpress.com TGGP

    Moreover, a futarchy will contain a third class of animal: bears, animals that are willing to lose money by making intentionally bad bets because they profit from bad policies.
    Robin mentioned manipulators, and considers them “sheep”. His contention is that they will attract wolves. He doesn’t even consider Bill Gates to be up to the task, so what source of “bears” are you considering?

    The accuracy of the market depends on the proportion of sheep to wolves
    Wrong. A Galton-esque “wisdom of crowds” aggregation consisting entirely of quite wrong (high standard deviation) sheep who are not even being rewarded for accuracy can result in a quite accurate prediction. The market will also develop over time so that if the initial conditions are biased toward some direction wolves can be attracted by the disparity. A quote from Robin’s futarchy paper: “To avoid letting such factors excessively influence policy, we can require that markets “clearly” favor a proposal over the status quo, via consistently higher prices over a substantial period of time.”

    does the informed bettor make a bet on what he actually believes the correct answer to be? or does he place a bet to make the aggregate more accurate?
    Are you familiar with Intrade or other betting markets. A bettor places a bet with positive expected returns. This means that placing a bet on the dark horse (even if that isn’t the bettor’s favored horse to win) will give more bang for buck.

  • Douglas Knight

    The wolves are consistently attracted to the sheep, and the net effect of having more wolves and more sheep is to have more accurate markets.

    Maybe, but the noise trader literature shows that this is not a slam-dunk. No, you don’t have to include every disclaimer, but I think this is important (though not relevant to MM), so I’m including it.

  • http://profile.typepad.com/robinhanson Robin Hanson

    Moldbug, the relevant task of speculators in a firm stock market changes every time the firm changes its policies or its competitive environment changes. Does this mean that such stock markets are always in their “initial state”? I’d say each new task is close enough to old tasks that they are most always in non-intial states. And speculators who deal with a series of similar policy proposals are in a similar situation, so that their experience with previous proposals is good enough. And your theory that “initial” markets statues have useless prices would seem easy to test in real financial market data – have you done such tests?

    On manipulators, I’ve done lab experiments (here and here), and others have looked at field data. Why don’t my experiments count as “initial” states? I’ve done models (here and here), which are of course is simplified – that is the point of models. Where are your models?

    Jeff and Dog, see these cites as well. I’m not proposing to implement futarchy full stop one day; I’m proposing to run a series of increasingly large trials. That would give you the data you want.

  • RP

    Most of this is completely irrelevant.

    According to Wikipedia, under futarchy, “elected officials define measures of national welfare and prediction markets are used to determine which policies will have the most positive effect”.

    The biggest, most obvious problem comes *before* the “and”. Most of this discussion is about what comes after it.

    The idea of a single, standardized measure of welfare, forcibly applied to a large population, is idiotic, whether it’s GDP, GHP, or anything else. It leads inevitably to disaster.

  • http://diogenes42.blogspot.com Diogenes

    Not an expert on this at all, but prima facie — futarchy is an extremely hard sell.

    Moldbug’s rant, whether correct or not, is rather amusing.

  • http://entitledtoanopinion.wordpress.com TGGP

    Abramowicz (author of “Predictocracy”) felt similarly to RP and so wanted to depend on legislators to determine after the fact whether policy was good or not. Hanson explained why he preferred just having them define GDP+ here. There is little reason to think that Congress will do a good job of that. Right now however they are both defining goals and deciding implementations.

  • Jeff

    Robin, I plan on looking at your models this weekend.

    Bears, as Dog named it, seem like something that would only appear in the system (if they will) after a certain threshold, and I wonder what that threshold would be.

    (I support increasing scope trials to acquire data on the subject. We need way more data)

  • http://profile.typepad.com/robinhanson Robin Hanson

    RP, futarchy can be applied at small scales, such as running firms using stock price as the outcome measure.

  • Steve Johnson

    “The wolves are consistently attracted to the sheep, and the net effect of having more wolves and more sheep is to have more accurate markets. Having a decision depend on a market induces people who want to manipulate the price, who are in effect sheep, since they attract wolves. You are somehow worried that decision markets that determine government policy would attract sheep without wolves, but I can’t imagine why.”

    Let’s say the policy in question is “Give $10 billion to Steve Johnson”.

    Unless the market is as large as the policy I can bid up the price of “It improves overall happiness” until I’ve spent almost all of the cost of the policy.

    How are the wolves going to feast on my “sheep” flesh in this case?

    Now imagine that the typical policies being evaluated in decision markets will be picked by someone with an interest in the outcome. Now imagine that people will game whatever measures you set up for measuring success of policy.

    “RP, futarchy can be applied at small scales, such as running firms using stock price as the outcome measure. ”

    Why would a firm do that when they could simply announce a policy and see how the actual stock price moves in anticipation?

    In contrast, in the internal prediction market you’ve got whomever plays in the small scale market evaluating the effects of corporate policy on stock prices. Guess what, people who are very good at this task exist and can be found. Look in hedge funds. Unfortunately, you’ll have to set up a larger market than the existing ones to lure them away.

  • http://unqualified-reservations.blogspot.com Moldbug

    Professor Hanson,

    Your sentences continue to be information-packed. Let me take them one by one:

    Moldbug, the relevant task of speculators in a firm stock market changes every time the firm changes its policies or its competitive environment changes. Does this mean that such stock markets are always in their “initial state”?

    An excellent question. It goes to the heart of the frequentist probability interpretation.

    Remember the Lakers-Nuggets game. What sample space is it in? The space of professional-sports games, the space of NBA games, the space of Lakers games, the space of Nuggets games, the space of playoff games. And of course it is also a one-time event: itself.

    Each of these events but the last consists of a set of different events. Patterns can appear in this set. A successful prediction algorithm

    Each sample space has its own wolf/sheep classification. A bettor may be an expert on the Lakers, for example, but know nothing about the Cavaliers. Her predictions may be reliable for playoff games, but not for regular-season games. She may understand the NBA, but know nothing about the NFL.

    As a non-abstract matter, for example, the “wolves” in a healthy financial market (I’m not convinced that ours is entirely in that category) are people who have some insight on the general task of mapping from the publicly-released accounting data of a corporation, to its future stream of returns. This is a sample space which is clearly defined and obviously generalizable across.

    If you send these wolves to Vegas and have them set odds for NBA games, you have a new initial state. Same if you have the bookies start a hedge fund.

    And speculators who deal with a series of similar policy proposals are in a similar situation, so that their experience with previous proposals is good enough.

    Sure. (Although of course it depends on what you mean by “policy” – this is a much broader sample space.) However, you still face the problem of training the market in the first place!

    And your theory that “initial” markets statues have useless prices would seem easy to test in real financial market data – have you done such tests?

    First, it’s not easy to test – because testing it would imply creating a new financial market. Moreover, our present financial markets have been trading more or less continuously for centuries, and any new financial market is a financial market, ie, in the same general sample space of the old. It would therefore attract the wolves of the old, rendering the market quite efficient and the experiment quite useless.

    But moreover, even if such a test could be performed, it would be entirely irrelevant to the analysis of futarchy. It may, or may not, be the case that some process of selecting initial participants in financial markets, or NBA markets, or jellybean markets, any other markets, will produce a population of sheep and wolves whose aggregate guess is good. But again, skill in one sample space is unrelated to skill in another sample space by definition.

    So, even if you could create a new financial market and you did, whether or not its initial aggregate guess was quite accurate, the result would imply no conclusion at all about the quality, distribution or motives of the initial participants in a policy decision market. Thus the experiment is useless.

  • http://unqualified-reservations.blogspot.com Moldbug

    Jeff, thank you for mentioning Schneier’s rule. It is certainly the best executive summary of my objection.

    The great irony of this conversation, of course, is that Professor Hanson’s futarchy paper consists almost entirely of a list of answered objections. Imagine if everyone played by these rules! His loaf has been in the oven, if only for half an hour at 200 degrees – the Buck Harkness of economics.

    Devin,

    You could start the decision market as a prediction market. At first people would bet on whether policy A will have results X. Once the wolves have eaten the sheep, and the market has learned to make very accurate predictions, you can then hook the market up to the machine and have the bets actually dictate the policy.

    It’s only fair to tell you that I thought of this. However, since I didn’t write it down, the credit is all yours. Professor Hanson may be in the same boat.

    Although your proposal is certainly an improvement on “cold-starting” a decision market, there are still two serious problems with it.

    One is that your “scratch-monkey” test market evolves wolves who can deal with sheep, but not necessarily wolves who can deal with bears. You might respond that a wolf is a wolf – an accurate predictor – who profits from mispredictors whatever their motivation. But if you look at Wall Street, you notice that feedback from the market’s present guess is a substantial part of many wolves’ algorithms. They look at both reports and charts. From the charts, a good wolf can decode the opinions of other wolves, the opinions of sheep, and the distribution of sheep.

    Furthermore, as Jeff points out, the wolves need to overpower the bears financially – they need to extract enough money from the bears to render any profitable interest in misprediction unprofitable. This is a matter of funding, not expertise. And a wolf in the prediction phase may turn into a bear in the decision phase, so you can’t just say the incumbent wolves will hold the financial high ground.

    Two is that it is much more difficult than Professor Hanson believes to test the accuracy of a market using historical information. To test the accuracy of a market is to search for trading strategies that would have profited against it. Easily done – but will your hindcast-tested strategies predict the future? You are surely familiar with the problem of data mining/dredging/snooping.

    The market is accurate if there is no legitimate, non-dredged algorithm that can beat it. But the problem of determining whether an algorithm is dredged (“Rodriguez hits .500 against white left-handed pitchers on Thursday nights”) or non-dredged (“Rodriguez can’t hit a fast sinker to save his life”), again, requires that fatal ingredient – actual human judgment.

    Thus you have no mechanical way of determining when your prediction market can be converted into a decision market, to begin its career in mechanical governance. Curses! Your quest for the philosopher’s stone, foiled yet again.

  • http://profile.typepad.com/robinhanson Robin Hanson

    Moldbug, you say every new financial market we have ever seen hasn’t really been different enough to see the effect you postulate, that “new” markets start out with a long learning period when they have useless prices. But somehow you think the markets I propose would be different enough to see this effect, and that it would be intolerable to suffer such a training period, even though markets can tell us how uncertain they are, and we could run the markets in an advisory-only capacity for as long as we wanted until we saw a clear enough track record to inspire our confidence in gradually relying on their advice more. We already have some relevant lab data, and we could try the concept first in smaller organizations before applying them to larger organizations. But apparently you think none of this could be good enough because we could never trust any track record about the past to predict anything about the future.

  • http://intellectual-detox.com Devin Finbarr

    Moldbug-

    All good points.

  • Yvain

    I like many of Moldbug’s points, but I do have a problem with the Emperor of China example.

    This blog has previously discussed probability distributions. If I ask you what my father’s name is, and you’ve never heard of him, “I don’t know” is a quick answer but not a technically complete one. You actually do know quite a bit about my father’s name: you know it’s more likely to be John than Cindy, and more likely to be either of those than Zeeeeeeeee. I could offer you a series of bets about my father’s name at different odds (would you take a bet where I give you a million dollars if my father’s name is Mike, and you give me one cent otherwise? What about a bet where you give me fifty cents if my father’s name is Zeeeeeeee, and I give you three dollars otherwise?) and we would discover that you actually had some quite strong opinions on how likely it was that my father’s name was various different things.

    Moldbug seems to be a frequentist, which makes this a little harder to explain, but hopefully he will agree that there is a certain probability distribution on which he stands to make the most money in a series of bets regarding my father’s name. That distribution will probably be something like the percentages of males born in my father’s generation who have various names: a set along the lines of [2% John, 1.5% Mike....001% Ezekiel...00000001% Zeeeeeeee.]

    We can come up with a similar distribution for any unknown question. For example, I am 99% confident that the Emperor of China has a nose less than a foot long, and almost 100% confident that he has a nose less than a mile long. My distribution for Gormball would be very close to 50-50, but if you offered me a fair bet in favor of Neptune and against Ganymede, I’d take it: Neptune’s so much bigger than Ganymede that its alien population would probably be much bigger (I wouldn’t have expected aliens to inhabit a gas giant, but this question seems to presuppose they can) and a higher population means more potential athletes and therefore a better team (the same reason I’d take a bet that the US could beat Andorra at any sport you choose). I wouldn’t jump at the chance to bet on Neptune’s superiority, but if I had to make a choice, I could honestly say I give them a >51% chance. We should have some probability distribution for all questions, even if it’s the boring case of “equal probability for all alternatives”.

    An ideal prediction market (and Moldbug has done a good job of proving that real world ones are not ideal) should be able to feed us a better probability distribution than any other method on all questions, including the Emperor of China’s nose question. Even if it feeds us an all-alternatives-equal distribution, we’ve still learned something useful; we’re now too smart to fall for someone saying the answer is definitely X and we can know that with certainty. The person who knows that there’s a 50% chance the market will rise tomorrow and a 50% chance it will fall tomorrow is better off than the guy who’s been tricked into believing some scam that claims they can predict the market for him.

    If all possible methods would do poorly (for example, the Emperor of China is by an astounding coincidence a mutant freak with a five mile long nose and no one could have anticipated this) then prediction markets will also do poorly. But assuming we do need a best guess on the Emperor’s nose length for some reason, then as far as I can see prediction markets won’t do any worse than anything else.

  • http://unenumerated.blogspot.com/2009/05/futarchy-experiment-wed-learn-great.html nick

    All useful prediction markets are also decision markets. Furthermore, Mencius and Robin are both ignoring a crucial distinction between futarchy and normal decision markets — in futarchy the decisions are coercive. More here.

  • Awake

    @RH, MM, respectively:

    “And your theory that “initial” markets statues have useless prices would seem easy to test in real financial market data – have you done such tests?

    First, it’s not easy to test – because testing it would imply creating a new financial market. Moreover, our present financial markets have been trading more or less continuously for centuries, and any new financial market is a financial market, ie, in the same general sample space of the old. It would therefore attract the wolves of the old, rendering the market quite efficient and the experiment quite useless.”

    The CDS market was pretty awful at calculating proper zero-sum insurance premiums for about its entire existence until the massive risk repricing of last year. Here you see a new product where not enough historical data exists to properly calculate a fair price – this plays to MM’s point, that such things take time to “train”- we never went through a true balance sheet/credit crisis until last year, and the CDS market was making idiotic predictions because it had never “learned” what happens in one of those.

  • josh

    Good for both of you for responding to one another (although Robin did attempt to be dismissive early on).

  • RP

    RP, futarchy can be applied at small scales, such as running firms using stock price as the outcome measure.

    Robin, are you saying you no longer advocate futarchy as a form of government, as you did here?

  • http://unqualified-reservations.blogspot.com Moldbug

    Yvain,

    The Emperor of China example is – to put it baldly – a bad one. It is meant to be a gedankenexperiment, as in Feynman’s original, but in this context it’s basically impossible not to read as a problem in physical anthropology. Yes, we do know something about the length distribution of Chinese noses.

    While I don’t see any errors in your reasoning, I think you are misleading yourself slightly by choosing overly trivial examples. Information about the distribution of names is also not hard to come by. When it comes to planetary gormball, or the number of beans in a hidden jar, or whatever, while we do have very little information, it is impossible to have none – we know the jar must fit under the desk, etc. But in this case, there is a much more subtle form of assistance – we know exactly what we don’t know.

    In prediction markets where information about the lack of information is easy to discover, it’s easy to be a wolf. Eg, in a coin-toss market, it’s easy to bet that the probability is 50-50. We don’t need a prediction market for this.

    In real-world problems – especially involving the future – “unknown unknowns” are routine. Consider the simple problem of computing the cumulative CPI (“inflation”) between 2010 and 2015. If you know the answer, you can make a lot of money. And the variable is easy to define in hindsight. But there are 17 different economic theories about the causality of consumer price inflation, and at least 16 of them are wrong.

    Difficult prediction problems generally involve separating a large number of spurious patterns from a real pattern. If there is a real pattern. Neither my examples of low-information problems (the Emperor of China, the jar under the desk) express this very well, and nor do your examples. Hopefully the real-world example above is clearer.

  • http://unqualified-reservations.blogspot.com Moldbug

    Moldbug, you say every new financial market we have ever seen hasn’t really been different enough to see the effect you postulate, that “new” markets start out with a long learning period when they have useless prices. But somehow you think the markets I propose would be different enough to see this effect…

    Yes, because the problem of analyzing corporate profits is substantially different from the problem of analyzing public-policy outcomes. How many people do you know who are expert in both?

    it would be intolerable to suffer such a training period, even though markets can tell us how uncertain they are, and we could run the markets in an advisory-only capacity for as long as we wanted until we saw a clear enough track record to inspire our confidence in gradually relying on their advice more.

    If I’d seen this proposal in your futarchy paper, I might have given you credit for it. Perhaps it’s there and I just overlooked it. But yes – see my two objections above.

    One: as soon as you switch the market from advisory to binding, the state of the market changes, because a “bear” strategy (losing money in the prediction market, to purchase decisions which make even more money) becomes viable. Nothing at all prevents a wolf from turning into a bear.

    Two: I think market validation is a lot more difficult than you think. Let me reason through a series of points, and you can tell me the first one you disagree with.

    A: To validate the efficiency of a market is to show that no counterexample trading strategy can be designed which profits against it.

    B: Given any historical record for any market, it is trivial to design a counterexample strategy which profits in hindsight.

    C: Most such profitable counter-strategies amount to data mining/snooping/dredging, and will not exceed random performance in future. A profitable counter-strategy is legitimate only if it profits across the entire sample space, ie the space of all possible markets – including future as well as past.

    D: Distinguishing between legitimate and illegitimate counterexamples to the claim that a market is efficient (a) requires an understanding of the problem that the market is solving (ie, the entire sample space, including both past counterfactuals and future hypotheticals), and (b) involves an assessment that is qualitative, not quantitative (data does not help).

    E: Therefore, to decide that the market is efficient, we must (a) prove a negative (no valid counter-strategies exist), and (b) must use qualitative assessments to filter out invalid counter-strategies.

    F: Therefore, the skill needed to decide whether to turn the market on demands expertise in the problem which is at least comparable to the expertise that would be required to do without the market at all, and instead make the decisions directly.

    Note, as a gut-check confirmation of this reasoning, that people still argue furiously over whether the stock markets are efficient or not. If there exists some methodology that could validate the efficiency of a market, you’d think it would have been deployed already.

    So, no: I haven’t seen your data. However, if I was shown your data, it would put me in the position of asking: who do I believe, (a) my brain or (b) your data? If I go with (a), I am at worst consistent in my error. Thus I have no logical interest in your data. If you are right and I am wrong, you have a better chance of explaining the error to my brain, than intimidating me with data.

    Alternatively, if you are willing to agree that there is an important qualitative distinction between a trained, adapted or iterated prediction market and an untrained one, this is my main substantive point. While, for the reasons above, I don’t feel that modifying futarchy according to Devin’s proposal would make it a good idea, I’m willing to admit that this improvement rescues it from “retarded.”

    And, of course, we’re in agreement that prediction markets are very good things and should be much more common than they are. However, I suspect that your private-sector customers would become more, rather than less, confident in the value of these devices if you erred more on the side of overstating their limitations. CEOs respond very well to underpromising and overdelivering.

    I suspect your readers would also appreciate (a) a comment on the vitality of the adversarial process in the modern university, and/or (b) a comment on the demand for mechanical decision procedures in government (ie, a response to Carlyle).

  • rafal

    Moldbug:

    “One: as soon as you switch the market from advisory to binding, the state of the market changes, because a “bear” strategy (losing money in the prediction market, to purchase decisions which make even more money) becomes viable. Nothing at all prevents a wolf from turning into a bear.”

    Note that if the decision is welfare-diminishing compared to the status quo, there will be a group of people losing more money than the bears gain. They would have a strong incentive to buy the decision back, wouldn’t they? And even if the decision is welfare-enhancing but sub-optimal, somebody fails to realize gains compared to the efficient decision, and has reasons to push towards optimal outcomes, in proportion to the deviation from optimality.

    As long as the values of various groups can be accounted for in a unified currency, making them fungible, a decision market is not that different from a prediction market.

  • qwert

    My two cents:

    Futarchy is useless because it essentially boils down to a newer weirder freakier form of American democracy. If the market is large and everyone can play and have a say, then this is just another election where people get the dumb crap they deserve. Imagine America going nuts betting over “Will gay marriage destroy our families?” There cannot possibly be wolves in such markets; only demagogues would gamble, since the outcome is nothing more than measuring who’s got more demagogues. This is like democracy. Sure, Professor Hanson would like us to believe that futarchy wouldn’t be deployed for such cheap stunts; but then again, Democracy was never supposed to be deployed for such cheap stunts either.

    If the decision concerns a small specific narrow interest, then the special interests will manipulate the market in order to get the policy decision they want. This is like lobbying.
    When the question of going to Iraq comes up, the people who care the most manage to bribe enough politicians, journalists and talking-heads to assure the war happens. In a futarchy, they’d spend this money on bending the market’s decision their way. Concentrated benefits and dispersed costs give the beneficiaries the ability to lobby and get their way. I see no reason why this would not translate to prediction markets. So long as the economic consequences of a decision exceed the market cap, then this is going to happen.

    And so, through votes and money influencing decisions, we would eventually pretty much get the same outcome as we do today with lobbyists and democracy.

    This exposes the problem with government decisions in general. No matter how great or accurate or wonderful or futuristic the decision-making process (dictatorship, democracy, monarchy, futarchy, populist) government decisions are a bad idea.

    And this is why libertarians like Dr. Hanson would make far more sense if they stuck to arguing for liberty and curtailing governments rather than trying to find more elaborate ways of squaring the circle of making government decisions intelligent and good. There is a perfect tried and tested way of utilizing the wonderful knowledge of markets to make people’s lives better: real life actual markets without government intervention.

  • http://profile.typepad.com/robinhanson Robin Hanson

    I rest my case against Molbug. The rest of you, please hold yourself to a higher standard than Molbug, and actually read what I’ve published about the manipulation problem (I linked to them
    above) before deciding I’m all wrong about that.

  • Yvain

    Thanks for your response, Mencius. Yes, inflation is a better example. But I still don’t understand why you’re critiquing prediction markets for being unable to predict really difficult questions well when traditional methods also fail and the markets might do a little better.

    Do you believe that traditional methods (like asking a famous economist) are consistently more likely to be right than ideal prediction markets? If so, why would a policy-maker be more likely to choose this brilliant economist than prediction market investors?

    Do you believe that there’s a high chance that even well-trained, highly-developed prediction markets will be far from ideal, and that it will be hard to detect when they’re far from ideal, and that therefore a famous economist can consistently beat a real prediction market if not an ideal one?

    Do you believe no one should ever try to predict difficult questions? Even in a libertarian society, in a few cases policy makers will just need some idea how likely certain things are. For example, the military will need to know what the chances are of getting involved in different types of wars over the next decade. This is a good example of one of your difficult problems where there are probably a lot of false signals, but the military can’t just shrug and give up and not plan for anything; it needs a best possible guess, even if it knows that guess has a huge margin of error. Someone somewhere in the Pentagon is probably working off of a vague belief like “X percent chance we’re in a land war with China over the next decade; Y percent chance we’re in a naval war.” Why not get X and Y from a prediction market instead of pulling it out of some general’s rectum? Neither will be all that good, but the market ought to at least be better.

    Or do you believe that prediction markets are as likely or more likely to be right than famous economists, but that policy-makers will have the good sense to understand that a famous economist is probably wrong, whereas they’ll trust a prediction market blindly? That’s a pretty reasonable point, but if education were at all possible it would be better than giving up on the whole idea forever.

    By the way, I like your blog. I think. I’m still too overwhelmed to really be sure.

  • Steve Johnson

    Do you believe that traditional methods (like asking a famous economist) are consistently more likely to be right than ideal prediction markets? If so, why would a policy-maker be more likely to choose this brilliant economist than prediction market investors?

    Famous economists are the result of a selection process. This process is one where accuracy is not the selected for characteristic. The whole field, in fact, isn’t one where an idea’s correspondence to reality is not what determines success or failure of that idea. As for why this is so, see MM’s points about academics play-fighting about inner circle ideas and real fighting versus ideas outside the inner circle.

    If the area is one where there is or a selection process for experts that actually works then, yes picking an expert and getting his opinion will be a better way of getting the answer right than asking a market. All a market is is a method for selecting experts and actually holding them accountable for results (so they can’t coast on reputation) by having bad results cost them money (which, in turn, can force them out of the market if they lose enough).

    As for why a policy maker would choose one method or the other, well, a policy maker will choose whichever method gives him the result he already wants. See public choice economics for the reasons why.

    Do you believe that there’s a high chance that even well-trained, highly-developed prediction markets will be far from ideal, and that it will be hard to detect when they’re far from ideal, and that therefore a famous economist can consistently beat a real prediction market if not an ideal one?

    The method by which markets make good predictions is by offering incentives (money) for experts to participate (trade). Ultimately, markets are only as good as actual experts are. If some things are completely non-predictable then markets will not successfully predict outcomes. Unfortunately, they don’t produce a result that says “this price is the result of people making guesses as to what other people think about the underlying”.

    Do you believe no one should ever try to predict difficult questions? … For example, the military will need to know what the chances are of getting involved in different types of wars over the next decade. …it needs a best possible guess, even if it knows that guess has a huge margin of error. Someone somewhere in the Pentagon is probably working off of a vague belief like “X percent chance we’re in a land war with China over the next decade; Y percent chance we’re in a naval war.” Why not get X and Y from a prediction market instead of pulling it out of some general’s rectum? Neither will be all that good, but the market ought to at least be better.

    1) What is the selection process for generals? Does it involve having a deep understanding of geopolitics? IOW, can you expect generals to be good at making this sort of prediction?
    2) Who benefits from preparing for a land war? Who benefits from preparing for a naval war? How big are these benefits? Let’s say you make anti-ship missiles and have the contract to supply them to the US Navy. How many more will you sell if the Navy gears up for a sea war with China? How much is it worth it to you to push the prediction market in your direction? Let’s say you’ve got connections with the Chinese communist party and you want to get into the Taiwan real estate business. How much is it worth it to you to make the US military weak enough so that they cannot credibly stop an invasion of Taiwan?

    In this case, you’re much better off trying to get the best possible method for selecting generals then letting them actually set policy in their area. There’s no magic that removes the need for human judgment.

  • http://profile.typepad.com/teageegeepea TGGP

    Ultimately, markets are only as good as actual experts are.
    Have you read What does a free market require? A market can function with mindless actors. Galton’s “wisdom of crowds” (though not actually a market) did not depend on any single individual having an accurate guess. Regarding political predictions, are you familiar with Tetlock’s “Expert Political Judgment”? The experts don’t do that well compared to a simulated monkey throwing darts at a board. Bryan Caplan gives a commendable defense of experts in his review of the book (which you can read here) but it’s mostly against a populist misinterpretation of Tetlock’s results. If you simply try to select “the best” experts you are probably going to wind up overpaying the tournament winners, and still have a small sample size (with differing opinions within the sample suppressed by group-think) subject to variance error and overconfident in its own predictions. Furthermore, selecting the best experts requires competence in the selectors to know expertise when they see it.

    If some things are completely non-predictable then markets will not successfully predict outcomes. Unfortunately, they don’t produce a result that says “this price is the result of people making guesses as to what other people think about the underlying”.
    As others have mentioned, in the emperor’s nose issue bettors will note the uncertainty by giving a wide probability distribution. Markets in which there is more confidence will concentrate probability in a narrower area.

  • http://unqualified-reservations.blogspot.com Moldbug

    Professor Hanson,

    I explained clearly why I’m not interested in your data. But do you really want a critique of your models? You seem awfully eager to get out of this conversation. Perhaps I should humor you in this.

    But I skimmed your first model paper (A Manipulator Can Aid Prediction Market Accuracy, which I must say is a remarkable title), ignoring the formulas and looking at the assumptions. It took me about five minutes to find the following assumption:

    Of the T traders, let us assume that N of them can acquire information about the true asset value v, and about the manipulator’s bias w.

    In other words, your “wolves” know not only the true value of the asset – they also get to see the trades of the “bears!” Words fail me. Well, actually, there’s one I can think of.

    Now, is this the only spherical cow in your model? Maybe it is, maybe it isn’t. But if it took me five minutes to find one, why should I look for more? And when blatantly unrealistic, self-serving assumptions of this type appear in your models, I can’t even imagine what beasties might be in your data.

  • Pingback: Should you be bullish on enterprise prediction markets? | Midas Oracle .ORG

  • Michael

    Of course, there are a variety of prediction markets already in existence. And, though I do not often quote the Bible, it is apt to point out that “ye shall know them by their fruits.”

    One that is particularly apposite to the present discussion is that in credit-default swaps. Continuing in the same mode, one is tempted to recall the passage “for now is the axe put unto the root of the trees, so that every tree that bringeth forth not good fruit is hewn down, and cast into the fire.”

  • Pingback: La patience a ses limites. | Midas Oracle .ORG

  • http://unqualified-reservations.blogspot.com Moldbug

    Yvain,

    All I can say is that if everyone approached prediction markets with the same skepticism as you, they would be truly useful tools. If they weren’t basically illegal, that is.

    There is no philosopher’s stone. Good government is government by good people. In essence, an iterated prediction market is just a very clever way to employ good people, in a very unusual organizational structure which is unusually tolerant to the presence of bad people in the population. (Not to mention unusually inexpensive from the management perspective.) Treat the tool as what it is, and you can’t go wrong. Treat it as a philosopher’s stone…

  • http://profile.typepad.com/robinhanson Robin Hanson

    Moldbug, you do not understand what you read. Our conversation is over.

  • http://unqualified-reservations.blogspot.com Moldbug

    rafal,

    Note that if the decision is welfare-diminishing compared to the status quo, there will be a group of people losing more money than the bears gain. They would have a strong incentive to buy the decision back, wouldn’t they?

    I think that’s the cleverest argument on this whole thread! Note, however, that this would create a kind of policy-control contest very familiar to, say, the citizens of the late Roman Republic. I’m not sure it passes Schneier’s test. Well, okay, I’m sure it doesn’t. On the other hand, that doesn’t mean it wouldn’t be an improvement on Washington’s present policy process…

  • Steve Johnson

    Note that if the decision is welfare-diminishing compared to the status quo, there will be a group of people losing more money than the bears gain. They would have a strong incentive to buy the decision back, wouldn’t they?

    They have the same incentive to buy the decision back as the consumers do to buy an open market in foreign sugar. Unfortunately, corn producers and sugar producers have stronger incentive to buy the market closed.

    Concentrated benefits, diffuse costs.

  • http://unqualified-reservations.blogspot.com Moldbug

    Well! I feel it’s only appropriate to give Professor Hanson the last word.

    Conveniently, I finished the aforementioned paper – and discovered that it concludes with an admirably-motivated, and quite plausibly complete, list of all its spherical cows. (In case it isn’t obvious, my objection above was to the last-named cow.) If this doesn’t place Professor Hanson in the 99th percentile of academic honesty, I myself am a spherical cow:

    Of course the fact that we have a particular model illustrating these results hardly implies that these results always hold in every context. Our model assumes risk-neutrality, normally distributed values and signal errors, interior choices of information quantity, only quantal-response-type irrationality, no meta-signals about the signals of other agents, no transaction costs of trading, no budget constraints, and a single rational manipulator with quadratic manipulation preferences and a commonly known strength of desire to manipulate. However convenient these assumptions may have been for solving the model, one can reasonably question the empirical relevance of models based on them.

    The prosecution rests – and calls for leniency.

  • Porphyrogenitus

    Whether they’re “accurate” depends in large part on definitions of accurate, and who controls those definitions. They’ll be different in decision markets (where accuracy depends on policy outcomes) than prediction markets (which simply predict, or not, what outcome will occur).

    We face this today and I do not believe we have *truly* objective criterion for evaluating whether the implementation of a policy on any subject of import the least bit controvercial was accurately predicted. That is to say, we already have a lot of data on policy implementations and outcomes, and it’s much harder to agree on whether their effects were accurately predicted in advance than with a market that simply predicts whether A or B will happen (“Will Barak Obama’s Stimulus Package pass and be signed into law” is a much simpler question than “what will the effect of Stimulous Package A be, will it be better than Stimulous Package B, and will either be better than doing nothing, and over what time frame, and with or without other ripple effects” – CBO’s predictions, for example, are considerabily different than the White House’s).

    In the abstract, people propose Policy A, others say Policy B will be better, some say the status quo is better than either. The advocates of Policy A say it will cost $100 Billion Woolongs and create 1.21 gigajobs. Critics say that it will cost four times that much in the end and produce half as many jobs.

    Policy A is enacted, it ends up costing several multiples of what was predicted and net job growth is undetectable. Advocates say it was because it was underfunded at the beginning, and that other factors impacted it, and given how the economy performed, conditions would have been worse without it: It saved jobs. Critics say their criticisms were accurate and that the policy made things worse ultimately, particularly because of unintended consequences (Brawndo consumption went up 400%, causing crop failures and an increase in illiteracy and out of wedlock childbirth, and these things aren’t included in the advocate’s data regarding whether the policy outcome was good or bad). Adocate’s rejoider is that well B, C, and D happened in the interval as well, and without Policy A in place, those things would have excacerbated the situation, so A really saved us from catastrophy.

    This is much more open to manipulation than a prediction market in that sense alone.

    Also, I think Hanson is well too dismissive of the ability of “Bears” to manipulate outcomes. In a world where George Soros & friends could cause a run on the Pound, it’s not hard to foresee that a subset of interested parties who will benefit disproportionately from the implementation of a policy will have more interest in investing enough to make sure it happens than everyone else. Will “Wolves” follow that money, because it is more likely to occur (and thus pay out), or will they vote with the “sheep” (those whose direct interest in the policy choice is less strong, and indeed who may pay the costs and receve no benefit)?

    “Smart Money” is more likely to go with the “Bears” on this, in my opinion. Their market behavior will be based on an evaluation of what will “win” rather than which will be the best policy to implement, and everyone else in the market will then be “sheep” (AKA the “marks”).

    For some reason a lot of the exchanges on “Wolves” vs. “Sheep” reminded me of Phil Helmuth going on rants about “idiots” who have offsuit Queen-Ten and raise. Phil’s “idiots” win hands, get to final tables, even win tournaments, despite the fact that though over time – if your time horizon is long enough – the pros do much better than “dead money”. When interest and passion is high, a lot of sheep enter the market who otherwise might not be there – and this will make the outcomes different than in the smaller-scale running experiments Hanson foresees as testing and building the case for Decision Markets.

    But in this situation the way the sheep will be separated from their money by the wolves is not directly tied to which policy outcome was accurately predicted to be preferable. The decision market will be dominated by those “Wolves” who are most skilled in knowing which decision will dominate, not which one is good policy and which one(s) are worse, and by players with an interest in the outcome (Bears) regardless of it’s overall impact (the “I’m All Right, Jack” set). The wolves will prosper in the market from investing correctly in which policy alternative will win, not which one will have the most benefitial impact.

    It “may” (to use Professor Hanson’s word) be that those don’t correlate at all. They “may” even be opposed. I’m not sure I’d bet my life that they will be alligned. I agree with Moldbug that there is no good way to test it – I have little confidence that “experimenting” in a small scale on inconsequential policy decisions won’t produce the same outcomes that going big on major choices will. Wolf money will tend to follow, rather than counterballance, Bear money, and everyone else by definition will become Sheep and get shorn.

    So in other words, no change.

  • Pingback: DYSPEPSIA GENERATION » Blog Archive » Futarchy considered retarded

  • Anthony

    Robin –

    your papers regarding the manipulator problem cover prediction markets where the rewards to the manipulators are on a similar order of magnitude as the stakes of the market players. In a futarchy model, this is likely to not be the case.

    As an example, imagine that the State of California has set up a decision market to decide whether to raise certain taxes or to cut certain programs/benefits/salaries to balance its budget in the forthcoming fiscal year. The payoff to the decision market players is based on whether the budget balances while avoiding certain disfavored results.

    Public-employee unions have, in general, a very large incentive to push the contract towards the “raise taxes+maintain spending” outcome. Very few businesses or individual taxpayers have similarly large incentives in the opposite direction, and based on prior history, it’s unlikely that a coalition would form which was willing to spend the amount of money that the unions would spend.

    To some extent, California’s initiative process provides a modest test of manipulation incentives, though there are some significant differences between propagandizing on initiatives and participating in a market.

  • http://hanson.gmu.edu Robin Hanson

    Anthony, you misread the models and experiments. Market speculators collectively have far deeper pockets than the California public employees union.

  • http://entitledtoanopinion.wordpress.com TGGP

    Let’s a imagine a policy in a subsidy-free decision market whose effects include transferring $X (an amount I already have or could raise) to me. I am indifferent about the other effects. I will then always be willing to bid up to $X (the point at which I am indifferent) in order to get the policy enacted. The guard against this is that other speculators will bid against me. If they are successful in moving the price back, the policy will not be enacted. Because my bet was conditional on the policy being enacted, my situation would then be unchanged. If the policy does get enacted then even if it fails to achieve the results I bet that it would, I still profit by X minus however much I bid (on the off-chance it does produce the results I bet on I receive X plus my winnings). Is all that correct?

  • http://www.bayesianinvestor.com Peter McCluskey

    To those who imagine something important happening when the market is “switched” from advisory to mechanically binding, think about the way a U.S. president is elected. Early presidential elections clearly didn’t mechanically translate popular votes into a certification of a victor. Has the electoral process become mechanical now? If so, when did it become mechanical? If not, how many people are actively working to make it more mechanical and do you have any reason to expect futarchists to expend effort making futarchy more mechanical than the electoral college currently is?

  • http://entitledtoanopinion.wordpress.com TGGP

    Peter, in Supercrunchers it is explained that experts assisted by prediction algorithms outperform experts by themselves. However, they underperform prediction algorithms alone. This is because people are sure the algorithm is wrong in certain instances and ignore its advice. In the reverse case where the algorithm has final decision power and is aware of an expert’s prediction, it outperforms the algorithm by itself. Hanson has said he wants to create a “fire-the-CEO” market, which is definitely a case where human discretion could lead to incorrect overriding. So I do think something important happens when it is made “mechanical” rather than merely advisory.

  • http://hanson.gmu.edu Robin Hanson

    TGGP, yes a manipulator’s payoff is the improvement in the decision induced by the distorted market estimates, minus trading losses from such distortions. And yes part of the reason to put a mechanism directly in control is to avoid losses from overconfident human corrections.

    Peter, yes it seems that the electoral college slowly and gradually became less important, so that national elections slowly became more mechanically in control of who is president. So there are degrees of direct control, and mechanical processes can slowly and gradually be placed in charge.

  • http://www.bayesianinvestor.com Peter McCluskey

    TGGP, when I try to imagine a realistic way in which “fire-the-CEO” markets become effective, what I imagine is that as the evidence mounts for the value of those markets, boards of directors that fail to follow the results of the market suffer gradually increasing risks of being sued for breach of fiduciary responsibility and/or voted out, and the quality of evidence they need to present to defend themselves against such actions keeps increasing.

    Also, I expect it will take some skill to read market prices and determine whether they are sending a signal or showing only noise. I expect boards of directors to initially have much discretion in making that distinction, and that as evidence grows the range of prices over which they can plausibly use this discretion will shrink, and will be gradually codified into rules.

    I have trouble identifying a specific point in these trends when the process becomes mechanical. Are you imagining something different, such as some software that sends out “you are fired” letters being turned on?

  • Pingback: Morons and Non-Morons — Sheep and Wolves | Midas Oracle .ORG

  • Pingback: ROBIN HANSON HAS RENOUNCED TO RULE THE UNIVERSE VIA PREDICTION MARKETS. | Midas Oracle .ORG

  • Pingback: Midas Oracle .ORG

  • http://freesoc.wordpress.com sconzey

    Do excuse the necro-posting, but I was startled to discover no-one’s even touched on what I feel is the weakest link in the futarchy proposal.

    First, let me just say that prediction/futures markets are awesome, and Hanson’s motivation — that thinking up effective and economic public policies ought to be profitable too is a laudable one.

    With that said, the whole system falls at the first hurdle, when we are required to vote on the way of evaluating the outcome of public policies. Not only does it assume that
    1. there is some objective “value function” for evaluating public policy, but it assumes that —
    2. whilst democratic vote is deemed a poor way of working out the outcomes of public policy, it is a good way to work out what those outcomes should be

    Assume, for instance, that I am a fundementalist Mormon. For me, an important aspect of public policy is that it minimises drug use and permits bigamy.

    I am a minority in the US, so my democratic votes are not going to have that much effect on the value-function. Whether the decision markets work or don’t, I’m going to be pretty dissatisfied with the outcome.

    However, if we abandon the idea of a universal value-function, we may also abandon the controversial decision market for predicting public policy outcomes. I “shop” for a legislature in the same way I shop for baked beans; by their outcomes. If a legislature’s public policy provides poor outcomes — intended or not — I will choose another.

    The exact method of “shopping” for a legislature may vary. Moldbug’s Patchwork, Friedman’s PDAs, or even the old-school Panarchy are all adequate solutions.

    This, in my opinion, is a far better way to make inventing good policy profitable.

    • http://entitledtoanopinion.wordpress.com TGGP

      Hanson has no problem with “shopping” for legal systems. Nor does he believe in an “objective” value function, he stated before his position on metaethics is moral anti-realism (same as mine) though he tried to work out a way to say that the moral realist consensus of philosophers wasn’t a disagreement. Michael Abramowitz made the same point as you about the possible errors behind defining GDP+ when arguing for Predictocracy (bet on laws we will retroactively approve of, which I think a bad idea since I don’t approve of many old but popular laws!) against Futarchy. I think his reason for voting on values is that he wants to “make a deal” with people who currently favor democracy.

      • http://freesoc.wordpress.com sconzey

        Ah, that makes perfect sense. I’ll try to track down his post on meta-ethics, thanks.

  • Pingback: A Gentle Introduction to Neoreaction (for Libertarians) | The Ümlaut