Development Futures

Over the last month I’ve sketched how decision markets might inform choices in marriage, college, college admissions, and medical treatment.  Today I consider development, i.e., advising governments and non-profits trying to improve "developing" nations and societies.  Rick Davies, Michael Strong (here and here), and Robert Hahn and Paul Tetlock have written already about this to varying degrees, but let me say it my way. 

Decision markets are markets where speculators set prices that estimate the consequences of a decision. The big win here would be decision markets estimating envisioned outcomes of proposed development projects.  Imagine that before the World Bank, IMF, or Gates Foundation approved a particular development project, we could learn of the discrete alternatives being considered, and of the measurable outcomes they planned to publish and use to evaluate success or failure.   

For example, there might be several Malaria-reduction projects being proposed, each of which will be evaluated later by the actual rate of Malaria in some area.  AIDS projects might be evaluated via later rates of infection.  Education projects might be evaluated later by literacy rates, while national investment or loan projects might be evaluated on growth rates of GDP.

Development futures could directly evaluate expected outcomes both given that a project was approved, and given that the project was not approved.  The difference between these numbers is the expected effect of the project.  Development futures could tell everyone that the Gates Foundation’s malaria project is not expected to change malaria rates, or that a World Bank loan to Mongolia would dramatically improve growth rates there. 

The cooperation of the Gates Foundation or the World Bank would not be required (beyond revealing projects considered and evaluation criteria).  Any other interested organization could create markets allowing anyone they approved to trade.  Ideally someone would subsidize cash markets where anyone could trade. 

Once development futures had built up a track record of unprecedented accuracy, they could cut through poisonous biases on both sides – to expose both wasteful projects built on wishful thinking, and insincere skepticism used as an excuse to do nothing.  Let’s find out what really works, and then let’s really do it. 

GD Star Rating
loading...
Tagged as:
Trackback URL:
  • http://profile.typekey.com/tobyord/ Toby Ord

    Sounds very good to me. Development features many of the world’s most cost effective opportunities, but very little thought goes into selecting the great rather than the merely good (or sometimes the actively harmful). Any systematic ways of improving the projects chosen have enormous potential, and this looks like a great example.

    I’ve been rather skeptical of some of the previous decision markets in this discussion (eg marriage and college markets) as they were very specific, attracting very few investors. Successful development is a big issue and the knowledge is widely distributed. This wouldn’t suffer from that scale problem.

  • http://www.givewell.net Holden

    Here are the missing ingredients necessary to make this work:

    1. Details of the development plan. I am amazed by how rarely it is possible to get descriptions of what aid projects literally do, in enough detail to be useful in any way for making the kinds of predictions you’re talking about.

    2. Outcomes data. This is also amazingly rare, especially on a region-by-region basis. You can’t bet on malaria infection rates unless you measure them; doing so is expensive (I’m in favor, but most aid projects skimp on evaluation due to a desire to “minimize overhead”).

    The fallacy of this post is a common one about decision markets: the idea that if we create the mechanism and incentive for people to predict X, people will predict X because people are smart. Being smart isn’t enough; you also need relevant information, and in the world of aid that’s often just too hard to come by (and changing that is a project in itself).

    I’m writing from the perspective of someone who is trying to “trade” these markets in a different way: find the charities best suited to improving people’s lives and donate to them. GiveWell is a public exploration of what charities do and whether it works – the hedge fund in this analogy, except that we publish everything we find on our public website because we’re not in this for profit. I believe we’re pretty much alone right now, and to me the priority is getting more information out there and making it *possible* to understand and bet on these issues, before we worry about giving people incentives to do so (especially since, if they’re altruistic and in a position to donate, the incentives and mechanisms for these bets are already there).

  • Carl Shulman

    Holden,

    Yes, often the later malaria deaths will never be measured at all. On the other hand, the MIT Poverty Action Lab is making contributions to the empirical evaluation of development projects via randomized experiments, and has made headway over the last few years with development entities.

    http://www.povertyactionlab.com/

    One could initially set up decision markets for the minority of projects that are already incorporating rigorous evaluation, and then set a trend if the demonstrations are effective.

  • http://www.givewell.net Holden

    Sure, but it’s a question of priority.

    I put it to you that the world is currently suffering from a massive deficit of information about what works in philanthropy. Most quality debate and analysis occurs over policy, not philanthropy (ironic, since it’s philanthropy where your opinion actually can translate directly into action). 99.9% of philanthropy discussion revolves around fundraising, or incredibly vague generalizations (I’m happy to walk you through that blog space if you’d like for some examples), and foundations and charities are incredibly (a) reluctant to measure; (b) secretive about what they do measure.

    GiveWell exists to change this, and it’s a battle – to get information from charities, to make sense of it to the point where it can be reasonably understood and discussed, to put pressure on foundations to open up and donors to care about more than pictures. That’s what I think we need to work on – getting the information out there. I think this battle is many times more worthwhile than the idea of creating an extra incentive (in addition to the existing one, altruism) to make predictions for the tiny portion of aid that we do know something about.

  • Michael Clemens

    1. The liquidity of the market would start out very very low. There are many thousands of projects each year and the large majority of the people who have the required information probably don’t have internet access. So it would have to be demonstrated on a handful of very large projects about which a large number of people with internet access were knowledgeable.

    2. What if a large fraction of the successes occur for reasons that are ex-ante unknowable? For example, if I had two contingent shares — one that paid $1 if Mount Vesuvius erupted next year and the average temperature in eThekwini, South Africa were over 70 degrees in the year 2050, and another that paid $1 if Vesuvius did not erupt and the average temperature in eThekwini were over 70 degrees — would these shares be bid to different values? How could they be? Empirically, of course, one could find that they are, just like one might find that Robin’s proposed shares get bid to different values.

    3. No one involved in the aid business has any reason to cooperate (see Lant Pritchett’s paper). But it couldn’t be done without their cooperation, since the method requires information on projects that were considered but ultimately rejected. Some sort of public shaming method would have to be used.

  • Mason

    I think we (most people reading this) can agree that decision markets are a good idea, and have diverse applications, from healthcare, and college to national defense and charity programs.

    They work because they reward people for sharing what they know and punish those who talk about topics they don’t know anything about.

    So what steps can we take to make decision markets happen? They are widely accepted in valuing companies, maybe marginal changes from there are a good place to start. This could include markets for CEO candidates, or office locations.

    An office location market may lead to an army base location market. A CEO market may pave the way for a presidential market.

    We can continue to go over different areas in which decision markets would be useful, but unless these areas are proposed and evaluated with the goal of finding the best market to start in, I’m not sure how much good it does to continue to outline the general idea to specific markets.

    Regarding a potential failure of decision markets in general;

    I am surprised at how ineffective markets are at discouraging corporate buyout and mergers. The buying company’s stock price almost always falls when buyouts are proposed, and hindsight generally confirms them to have been poor investments, yet they continue.

  • april

    I don’t wish to to minimize Holden’s very valid points about the paucity of information on development projects. However, for every project the World Bank prepares, it does release both a Project Information Document (during the elaboration of a proposed project) and a Project Appraisal Document (when it is a approved) – which contain the key elements of what activities the project will support, and what goals (intermediate and outcome) they are targeting.

    I agree with Michael (and Lant Pritchett) though – for any additional information which the bi-laterals or multi-lateral development institutions would need to make public in order to inform the bets in the decision markets – some serious external pressure would need to be placed on them.

  • http://profile.typekey.com/robinhanson/ Robin Hanson

    Toby, decision markets can work with only a few investors.

    Holden, for most regions we should at least be able to get aggregate mortality rates and economic growth rates.

    Holden and April, giving more info should help markets make better estimates, but they will make estimate with whatever info they have. If other evaluation institutions do not have better info, then decision market estimates can be competitive with those other institutions. And with enough of a market subsidy, people from inside these organizations can be tempted to trade secretly to reveal their info.

    Michael, yes of course you’d want to start on the largest projects, but with enough of a trading subsidy the markets could in fact estimate thousands of projects. If you see prices you think irrational you would be paid to correct them.

    Mason, we are trying everywhere we can.

  • http://thesaifhouse.wordpress.com saifedean

    This sounds like a very interesting idea to me, but I do have one concern: the places where development projects are least likely to succeed are the places that need these projects the most; whereas the places where the impact is likely highest are the places where these projects are not needed the most.

    My concern here is that the high expected positive returns that will be apparent from (presumably) accurate forecasts will drive more and more investment towards areas that do not need this investment that much, and away from the areas that need it the most.

    Even though we would get a higher impact in the relatively well-off places, morally, a development organization has an obligation to concentrate on the less well-off areas, where even a smaller impact of the project might be much more important in humanitarian terms in the long run.

    This again goes back to the problem of aggregating across differnt impacts and measuring them compared to one another: health, education, income, etc…

  • http://profile.typekey.com/robinhanson/ Robin Hanson

    Saife, I don’t see how a mechanism for finding out the size of a project’s impact can be blamed for not taking into account the value of that impact. The markets are just intended to tell you what projects do what; it is a separate question to ask what effects have what value.

  • Shakespeare’s Fool

    Robin,

    Thank you.

    For many years “Mrs. Shakespeare” and I have been
    looking for a source of information on the effectiveness
    of charities. The BBB, Charity Navigator, and other
    evaluators have been of modest help. Comments by Peter
    Drucker and Tom Peters have been helpful in other ways.
    But we have never before come across anyone who was trying
    to evaluate charities on their effectiveness in as
    systematic and thorough a way as Holden’s GiveWell.
    How useful they will be, I do not know and at so early
    a stage in their efforts would not even try to guess.
    Despite their having put the bulk of their information
    in the user-hostile PDF format, the first half was well
    worth the read. Their work may be worth following and
    perhaps a little volunteer time.

    So thank you for your work on Decision Markets
    which led to your post that brought GiveWell
    to our attention.

    John

  • douglas

    Holden– My wife and I give to charities. While what Robin is talking about maybe of interest to some, what you are doing would be much more valuable to me.
    Keep up the good work!

  • http://www.givewell.net Holden

    John and Douglas – great to hear that you guys are interested in GiveWell. One of the big questions about our project is who’s out there who would use it, and how we can serve them better. So please take a good look, fill out our survey if you get a chance, and email me with any comments.

    Robin – yes, humans can bet no matter what information they have. But having less information leads to wider markets and less potential for value-added over the project funder. When we’re talking about betting on the mortality rate across an entire country 3 years from now, that’s a low-ratio bet even when the expected value is high. Nobody disagrees with your conceptual point, the question is where the priority should be, and I think it needs to be on getting information to people who already want to use it rather than paying people to bet on information they have very little of.

    As for subsidizing insiders to trade on what they know, I oppose this idea outright, because I’d rather pressure funders to reveal their *actual information*, and let others get in on helping to interpret it, than incent them specifically to continue hiding it so they can try to take advantage of gamblers. Which do you think is more valuable to a group of intelligent, analytical individuals: the foundation’s numerical prediction re: mortality rates, or the information that goes into it?

  • http://profile.typekey.com/robinhanson/ Robin Hanson

    Holden, I certainly don’t oppose getting these orgs to officially reveal more about their projects. But imagine an org that privately knew its projects had little chance of success, and whose donors who were giving the org the benefit of the doubt. If outsiders could discern this problem even without extra access to insider info, markets could publicize this failure and embarrass the org in a way donors would find it harder to ignore.

  • http://www.givewell.net Holden

    How often do you think an insider privately “knows the project will fail?” Dealing with people in this sector, one consistent pattern I’ve observed is a faith in the value of the organization and its activities that borders on religious. Of course, talking yourself into the value of what you’re doing is common; what you’re describing seems much less so. What is the profile of someone who is involved with and close enough to a program to be able to see that it won’t work, yet not influential enough to do something about it, and mercenary enough to continue with the organization because of their paycheck (generally below-market in this sector anyway) but disclose the information on the sly in the hopes of ripping off someone on a gambling exchange? I don’t think there are a lot of those floating around.

    I don’t think the problem is that people are promoting what they know are futile projects. I think the problem is that people aren’t evaluating and discussing the projects critically enough.

  • http://profile.typekey.com/robinhanson/ Robin Hanson

    Holden, I meant to describe insiders who had enough info to, if they were honest with themselves, realize the project had little chance of much success. Yes, of course, they usually aren’t honest with themselves. That is why it could be good to include traders who do have strong incentives to be honest with themselves.

  • Mason

    Holden,

    Any org that didn’t make project information readily available to traders would automatically make people question its usefulness. Orgs that engage in successful useful projects at reasonable costs will want everyone to know.

    This of course is currently the case, successful projects are publicized and unsuccessful ones are hushed up. So as with companies and stocks people can base donations to charities on past performance. This is where GiveWell is doing a great job.

    But decision markets help ensure the success of future projects. The market is much better than the individual at gathering and assessing information.

    Robin’s examples (malaria nets and country loans) of were both for future projects. As you know we don’t have unlimited resources so we have to decide which projects to support and markets make these decisions much better than individuals.

  • http://www.givewell.net Holden

    “Any org that didn’t make project information readily available to traders would automatically make people question its usefulness. Orgs that engage in successful useful projects at reasonable costs will want everyone to know.”

    I am telling you that this is false. Fundraising today is done either with emotional appeals with zero relation to facts, or with such liberal use of “analysis” as to exaggerate the returns on charity beyond what they could reasonably be. That means there’s no incentive to share an actual success, because if you give all the details, it will look like an abject failure next to the un-filled-in stories other orgs tell. For a precise example of what I mean, see this report on microfinance < http://research.yale.edu/karlan/deankarlan/downloads/WSJDefenseUsury.pdf> vs. this one < http://www.opportunity.org/NETCOMMUNITY/Page.aspx?pid=217&srcid=193>. The former can’t compete, in fundraising terms, with the latter.

    “Markets make decisions better than individuals” is often true, but it is not a universal unconditional truth. It depends on the things that often make it true being true, including: traders have access to enough information to form acceptably narrow bid-offer spreads (not currently true in charity), and traders have enough incentive to bet (not true when the return-*to-risk* ratio is unacceptably low, as it is when so little info is available and anyone with an acceptably narrow bid-ask spread is likely to be an insider).

    We are trying to change these circumstances. As I’ve said, there already is a mechanism for betting on projects (donating), so the problem isn’t there isn’t a market, it’s that the market’s broken.

  • http://www.givewell.net Holden
  • Peter Saintonge

    You could sorta kinda get to this today, by bundling futures or ETF’s for a given country together with development projects. Some projects could be so useful (say, cleaning up commercial courts) that they could, in theory, be self-funding if bundled with, say, options on ETF’s for that country.

  • Pingback: Decision markets are markets where speculators set prices that estimate the consequences of a decision. | Midas Oracle .ORG