Tag Archives: Prediction Markets

Risk Rating Reluctance

Managers of many financial organizations are arguably tempted to take too much risk with organizational investments.  Reputation-wise, such managers often gain more from stellar investments than they lose from disastrous ones.  Many regulations try to address this problem by limiting such organizations to “safe” investments, as determined by a few official ratings agencies.  This creates a demand by such managers for assets that are actually risky, but which are officially rated as safe.

Apparently the recent financial crisis was due in part to those official risk rating agencies supplying this demand; they are private firms and profited from applying too little skepticism to whether certain key assets really were as safe as some claimed.  No doubt those agencies say that this was all a terrible accident, that they never intended to profit from mistakenly calling risky things safe, but since they seem to have suffered little from the harm they assisted in creating for others, I have my doubts.

In all the financial reform discussions, I haven’t heard any proposals to address this specific problem.  And I’ve always wondered why we need investment ratings agencies anyway.  In principle, the right financial assets can give any elements you like from a full joint probability distribution over asset returns; how could a ratings agency expect to do consistently better?

At lunch today I asked my wise colleague Garett Jones about this, and he suggested that big financial orgs like being rated by people they can pressure. If they don’t like a rating, they can work their elite-school-alum networks of contacts to apply pressure to the ratings agencies to change unwanted ratings.  Such pressure is much less effective on financial market prices.  Garett also pointed me to this passage of his:

[Remember] the debate over subordinated debt in the early 2000’s, surveyed in Stern and Feldman’s Too Big to Fail. The Gramm-Leach-Bliley financial reform bill attempted to create a class of subordinated debt that would be explicitly banned from any future bailouts.  Major financial institutions would have been be required to hold some portion of their liabilities in the form of subordinated debt in order to give financial markets and regulators alike a market-based measure of firm health: If yields on a major firm’s subordinated debt spiked, that would be a warning sign.  But financial institutions and the Federal Reserve Board both pushed back against this market-based indicator, and so the subordinated debt requirement never made it through the regulatory process. … Firms will resist issuing debt that is bailout-free, and will overwhelmingly prefer debt that is bailout-qualified.

Bailouts mess up the connection between asset prices and their inherent risks.  Ordinarily, market prices only tell you the chance that a debt will be paid, not whether it would have been paid without a bailout.

This is all moderately bad news for prediction markets in such firms.  Apparently well-connected managers already know they prefer estimates by officials who respond to social pressure, over hard-to-manipulate market estimates, even if the later are more accurate.  Of course less well-connected managers should prefer the opposite, but who wants to signal their bad connections by endorsing independent markets?

Added 8a: Unnamed points us to Matt Yglesias responding in Sept. to Kevin Drum and an August Policy Report by Mark Calabria.  Kevin says:

Over the past decade ratings agencies were, at best, negligent, and at worst, perpetrators of outright fraud.

but doesn’t think raters were a big problem because other orgs used similar risk models and both buyers and sellers liked the mis-labeling.  Yet this is just what a corrupted regulator model predicts.  These folks consider switching who pays the raters, or making them a direct government agency, but not replacing them with direct market price risk estimates – why so blind to such an obvious solution?

Weighing Scientists

The latest (top science journal) Nature has an editorial on the need for better ways to communicate expert uncertainty on key topics like climate change, and a two-pager by Willy Aspinall on “More tractable expert advice“:

Of the many ways of gathering advice from experts, the Cooke method is, in my view, the most effective when data are sparse, unreliable or unobtainable. … Take as an example an elicitation I conducted in 2003, to estimate the strength of the thousands of small, old earth dams in the United Kingdom. Acting as facilitator, I first organized a discussion between a group of selected experts. … The experts were then asked individually to give their own opinion of the time-to-failure in a specific type of dam, once such leakage starts. They answered with both a best estimate and a ‘credible interval’, for which they thought there was only a 10% chance that the true answer was higher or lower. Continue Reading "Weighing Scientists" »

My Moldbug Debate

Mencius Moldbug and I did debate futarchy Saturday.  I’m not sure when the vid will be posted; here’s audio (start at 1:30).  Moldbug also has a follow-on 3700 word post; his main complaint is manipulator payola:

Suppose player P stands to make $X from decision D. In our chess example, he might have a side bet, paying $X, that White will open with P-KB4. Therefore, the question is: what will be his expected loss, $Y, from buying enough P-KB4 bets that White opens with P-KB4?  If X is greater than Y, manipulating the market (ie, moving it intentionally) is profitable. …

Professor Hanson … addresses this problem in three ways.  First, he constructs a model in which Y is infinite and X is finite. … Second, he does sociological experiments with undergraduates. He sets up these markets such that Y is greater than X. …  Third, he finds actual markets in which actual manipulations fail. …  None of this goes even a millimeter toward proving what needs to be proved – namely, that in all markets, Y is always greater than X. …

Honestly, I think the greatest difference between my perspective and Professor Hanson’s is just that I have much higher standards. His entire argument proceeds from the position that, since government today is so bad, anything that could be somewhat less bad is worth a look. … Don’t people buy decisions now? Well, gee, they sure do. So there you go.

For me, government safety is like airplane safety. Not only do I want a watertight proof that Y is greater than X, I want two or three parallel and independent proofs. At least one of them will probably turn out to be wrong. Professor Hanson is a professor, and thinks like a professor. I’m an engineer, and think like an engineer.

I am also a student of history. … the European writers of the Victorian era, whose aristocratic governments worked much better than ours, and were thus appalled by government failures which to us seem trivial and not worth mentioning.

In the debate, I suggested we start by trying fire-the-CEO markets, and only gradually rely more on them in CEO decisions as such markets collect good track records.  Moldbug seems to accept wide trading in ordinary stock markets because he doesn’t think any decisions depend on them, but strongly advises against allowing non-employees to trade in fire-the-CEO markets, due to manipulation concerns.  But even a track record showing that firms which followed market advice do better on average than firms that do not would not persuade him.

In fact, Moldbug the “engineer” says no data anyone could collect in the lab or in any organization smaller than a nation would be relevant, and even with nations he doubts we’d see hidden manipulation. Nor does any data collected in the last century test his belief that the best governments are single rulers running city-sized polities with iron fists and complete discretion.  It is not even clear what prior data makes his case – apparently it can’t be summarized in any concise form; you have to just read dozens of books and have a feel for it.

Not only does Moldbug know such iron fists would rule best, allow emigration, not cheat their investors, and never ever accept manipulator payola, he apparently knows this deductively, as a noble philosopher, not like data-addicted corrupt pansy social scientists.  And he has no interest in improvements in the status quo below his philosopher-deduced-best pinnacle.

What more can one say to such a person?

Real Rationality

Bayesian probability is a great model of rationality that gets lots of important things right, but there are two ways in which its simple version, the one that comes most easily to mind, is extremely misleading.

One way is that it is too easy to assume that all our thoughts are conscious – in fact we are aware of only a tiny fraction of what goes on in our minds, perhaps only one part in a thousand. We have to deal with not only “running on error-prone hardware”, but worse, relying on purposely misleading inputs. Our subconscious often makes coordinated efforts to mislead us on particular topics.

But at least many folks are aware of and try to deal with this; for example, I’ve seen a lot of good related posts on this at Less Wrong lately. There is, however an even bigger way in which the simple Bayesian model is extremely misleading, and I’ve seen no discussion of it at Less Wrong. We may see one part in a thousand of our minds, but that fraction pales by comparison to the fact that we are each only one part in seven billion of living humanity.

Taking this fact seriously requires even bigger changes to how we think about rationality. OK, we don’t need to consider it for topics that only we can influence. But for most interesting important topics, it matters far more what the entire world does than what we personally do. For such topics, rationality consists mainly in the world having and using good systems (academia, news media, wikipedia, prediction markets, etc.) for generating and distributing reliable beliefs on which everyone can act.

When seven billion minds are involved, the overwhelming consideration must be managing a division of labor, so that we don’t each have to redo the same work. Together we must manage systems for deciding who should be heard on what. Given such systems, each of us will make our strongest contributions, by far, by fitting into these systems.

So to promote rationality on interesting important topics, your overwhelming consideration simply must be: on what topics will the world’s systems for deciding who to hear on what listen substantially to you?   Your efforts to ponder and make progress will be largely wasted if you focus on topics where none of the world’s “who to hear on what” systems rate you as someone worth hearing.  You must not only find something worth saying, but also something that will be heard.

Yes, existing who-to-hear systems are far from perfect, but that fact simply does not make it rational for you to work on topics where a better system would approve you, if only such systems existed. Wishes are not horses. It might make sense for you to work on reforming our systems, but even then your best efforts will work through channels where current systems can rate you as a person to hear on that meta topic.

When what matters is how the world acts, not how you act, rationality on your part consists mainly in improving the rationality of the world’s beliefs, as determined by its main systems for deciding who to believe about what.  Just wishing we had other systems, or acting as if we had them, is delusion, not rationality.

From a conversation with Steve Rayhawk.

States Sting Status

We usually take control as a strong marker of status; those who give orders have higher status than those who take orders.  So, for example, bosses are reluctant to oversee better paid subordinates, and teens chafe under the control of their parents and teachers, even when their lives are otherwise comfortable.
People care about the form of government they live under not only because different forms of government have different chances of leading to peace, prosperity, etc.  People also care about how governments more directly influences their status.  For example, in addition to or setting aside our beliefs about which forms of government lead to which other outcomes, I suspect most of us prefer:
democracy to autarchy, as it gives us more illusion of control.
proportional representation, as gives more control over the person we pick
equal votes per person, as otherwise others have more votes than you
the state to be controlled by a group we identify with, so we seem in control
stigma be attached to welfare given to groups we don’t identify with
more regulation of competing high status, to bring them down to us
more support of affiliated high status, to bring us up with as they rise
laws not treat us like children or fools, as that degrades us
I suspect such status issues drive our actual choice of government forms more often than we like to admit.
Thinking along these lines, I was wondering about the status effects of something like futarchy — what if every time the government considered a policy, you had the option to bet for or against that policy, and such bets influenced policy?
Yes, you might still have to suffer the status-reducing indignity of being ruled by foolish policies chosen by clueless folks who in a just world would be considered your inferiors.  But you would always know that you had the option to have a large influence, via bets, on those policies, an influence far out of proportion to your fraction of the population.  You would also know that you could, via bets, arrange to be paid lots of money when those policies went badly, just as you had predicted.  Would this raise your status, relative to only influencing policy via your tiny fractional vote, and then just having to live with the consequences?
Setting aside whether this betting system would actually choose good policies producing peace, prosperity, etc., the question I’m asking in this post is if this betting system might substantially shrink the status sting of the state.  Yes this would not fully assuage a libertarian’s outrage at being subject to policies he did not (recently) choose, but would it be a substantial step in that direction?

We usually see control as a marker of status; those who give orders have higher status than those who take orders.  So, for example, bosses are reluctant to oversee better-paid subordinates, and teens chafe under the control of parents and teachers, even when their lives are otherwise comfortable.  People also hate or love their governments in part because how it makes them feel controlled by others, or in control of others.

More generally, people care about the governments they live under not only because different types of government have different chances of leading to peace, prosperity, etc. People also care about how governments more directly influence their status. For example, in addition to wanting governments that induce other outcomes like peace or prosperity, I suspect most of us prefer:

  1. governments with forms like those of recent high status regimes,
  2. to be part of large rich powerful empires, since those are high status,
  3. democracy over autarchy, as it gives us more illusion of control,
  4. proportional representation, as we then more control who represents us,
  5. equal votes per person, as otherwise others have more votes than us,
  6. states controlled by groups we identify with, so we seem in control,
  7. stigma attached to assistance given groups we don’t identify with,
  8. more regulation of competing high status folks, to bring them down to us,
  9. more support of affiliated high status folks, to lift us as they rise, and
  10. laws that treat them but not us like children, as that degrades folks.

Such status issues may drive our choice of government forms more often than we like to admit.  So when trying to design good government, we need to take such status affects into account, so that our designs can be attractive and stable.  Thinking along these lines, I was wondering about the status effects of something like futarchy — what if every time the government considered a policy, you had the option to bet for or against that policy, and such bets influenced policy?

Yes, you might still have to suffer the status-reducing indignity of being ruled by foolish policies chosen by dimwits who in a just world would be considered your inferiors. But you would always know that, via bets, you had the option of a large influence on those policies, far out of proportion to your fraction of the population.  You would also know that you could, via bets, arrange to be paid lots when those policies went badly, just as you had predicted.  Would this raise your status, relative to only influencing policy via your tiny fractional vote, and then just having to live with the consequences?

Setting aside whether this betting system would actually choose good policies producing peace, prosperity, etc., the question I’m asking in this post is if this betting system might substantially shrink the status sting of the state.  Yes this would not fully assuage a libertarian’s outrage at being subject to policies he did not (recently) choose, but would it be a substantial step in that direction?

Hewitt on Futarchy

Chris Masse taunted, “If he had balls, Robin Hanson would debate Paul Hewitt, instead” of Mencius Moldbug.  Monday Paul posted a 7000+ word critique of futarchy. I commented, “Care to indicate the top three claims you’d most prefer I respond to?” Paul listed three, and Eric Crampton responded much as I would. But since Paul probably wants to hear it from me, here are those claims, and related (long) quotes from Paul’s post: Continue Reading "Hewitt on Futarchy" »

Forum = Meta-Method

  1. How to pick city policies, vs. how to pick the mayor.
  2. How to cook a meal, vs. how to pick a restaurant.
  3. How to win a game, vs. how to decide which team won.
  4. How to do a study, vs. how to pick a study to publish.

These are four examples of methods vs. forums.  Methods are ways to do things; forums are ways to pick who decides what to do.  Yes, in a sense forums are methods, since choosing who decides indirectly picks what to do.  But that is what makes forums powerful; good forums induce people to find good methods.  Good  elections induces good city policies, good restaurant competition induces good cooking, good game rules induce good play, and good journal review induces good articles.

To me, prediction markets are mostly interesting as forums, not methods.  Alas many seem to confuse the two.  E.g., Ian Ayres at Freakonomics:

One of the great unresolved questions of predictive analytics is trying to figure out when prediction markets will produce better predictions than good old-fashion mining of historic data. … We are about to have a test of these two competing approaches … a cool Supreme Court fantasy league, where anybody can make predictions about how Supreme Court justices will vote on particular cases. …

[Will aggregate] predictions of the league [be] more accurate than the predictions of a statistical algorithm developed by [five stat experts?] … The fantasy league predictions would probably be more accurate if market participants had to actually put their money behind their predictions. … Statistical predictions could probably be improved if they relied on more recent data and controlled for more variables.

More meta-methodological comparisons like these … will also shed light on whether market participants will learn to efficiently incorporate the results of statistical prediction into their own assessments. At the moment, individual decision-makers tend to improve their prediction when given statistical aids; but they still tend to wave off the statistical prediction too often.

James Surowiecki’s book seems responsible for so many folks equating “prediction markets” with “wisdom of crowd” averages of non-expert more-intuitive opinion, vs. formal expert analysis.  Averaging popular opinion may be an interesting method, as is statistical analysis, but comparing these does not evaluate prediction markets as forums.

“Prediction markets” started from speculative markets, e.g. stocks, where accuracy comes much less from non-expert participation and much more from participants with incentives to self-select as experts.  Any team that considers itself expert enough can pay to prove itself, but in fact most teams stay away and prices tend to be dominated by real experts, who get paid and really know better than most.

Prediction markets aren’t about emphasizing ordinary Joes over credentialed bigshots; they are about emphasizing whomever tends to be right.  Simple opinion averages maybe be reasonable indicators of crowd wisdom, but they have too little of the forum-ness of letting self-selected expert teams come to dominate.

It seems to me that when academics like Aryes call for academic studies of prediction markets as methods, instead of as forums, they are implicitly suggesting that current academic institutions should be the forum in we choose forecasting methods.  If academic journals prefer a method, they suggest, that’s the method the world should use.

In contrast, I suggest prediction markets may be a better forum than academic journals for choosing forecasting methods.  Maybe the world shouldn’t use a method just because academics say its great; maybe those impressed with a method should have to put their money where their mouth is and trade on that method’s forecasts in prediction markets.  Maybe the rest of us should just accept prediction market prices as our best estimates; if and when prediction market prices become dominated by traders using a method, that is when the rest of us will have implicitly accepted that method as best.

Rah Price Manipulators

Matt Yglasias complains Climate Change Futures Markets would be manipulated:

This idea has some merit, but let’s not get carried away with ourselves. The underlying intuition here is that talk about climate change is cheap, but if we made people put their money where their mouth is we’d force them to speak honestly. The problem is that when coal and oil interests or the Koch family pays people money to mislead people about climate science or clean energy policies they are putting their money where their mouth is. Big money is at stake in this issue, and it could be easily worthwhile for polluters to lose money on a prediction market if that helped undercut support for clean energy legislation.  The problem is that just about any metric you might like becomes contaminated once people know there are large political economy stakes.

No!  Some metrics are more corruptible than others, and prediction market prices are especially incorruptible.  In fact, big money manipulators with legislative agendas would be good for climate change futures markets!  If most anyone can play, we expect a real money prediction market to get more accurate as more big money powers are known to want to manipulate them.

We have explained the mechanism in a 2009 Economica theory article, and confirmed its predictions in two lab experiment articles, one published in JEBO in 2006. Here is a summary: Continue Reading "Rah Price Manipulators" »

It’s News On Academia, Not Climate

Electronic files that were stolen from a prominent climate research center and made public last week provide a rare glimpse into the behind-the-scenes battle to shape the public perception of global warming.  …

“I can’t see either of these papers being in the next IPCC report,” Jones writes. “Kevin and I will keep them out somehow — even if we have to redefine what the peer-review literature is!”  In another, Jones and Mann discuss how they can pressure an academic journal not to accept the work of climate skeptics with whom they disagree.. … “I will be emailing the journal to tell them I’m having nothing more to do with it until they rid themselves of this troublesome editor.” …

Horner … [said] the e-mails have “the makings of a very big” scandal. “Imagine this sort of news coming in the field of AIDS research,” he added. … some likening the disclosure to the release of the Pentagon Papers during Vietnam.

More here.  Joel Achenbach comments:

This is not a scandal so much as a window on real scientists working on a politicized issue. … “Gravity isn’t a useful theory because Newton was a nice person.” I agree. But isn’t it also true that Newtons antipathy towards Hooke and his use of his position in control of the Royal Society, ensured that the concept of an achromatic lens for a telescope … had to wait until after [Newton's] death.

Yup, this behavior has long been typical when academics form competing groups, whether the public hears about such groups or not.  If you knew how academia worked, this news would not surprise you nor change your opinions on global warming.  I’ve never done this stuff, and I’d like to think I wouldn’t, but that is cheap talk since I haven’t had the opportunity.  This works as a “scandal” only because of academia’s overly idealistic public image.

It is a shame that academia works this way, and an academia where this stuff didn’t happen would probably be more accurate.  But even our flawed academic consensus is usually more accurate than its contrarians, and it is hard to find reliable cheap indicators saying when contrarians are more likely to be right.

If you don’t like this state of affairs join me in trying to develop a more reliable consensus mechanism on such topics: prediction markets. It just takes time or money.  Prefer instead to act shocked, just shocked, when the other side is shown to do this stuff, while reserving your side’s ability to do the same?  Then I have little respect for you.

Added 23Nov:  Tyler basically agrees.  Bryan too, mostly.  Nate Silver riffs.

Combo Prediction Markets Talk

I talked Friday at prediction parkets summit, on “Combinatorial Prediction Markets.”  Here are slides and audio.