Reward Robustness

Me in October:

In August I reported that economic disasters seem thin-tailed, and so are not existential risks.  Even so, [a new study suggests] we should still devote more attention to them.

What makes the global economy vulnerable to economic collapse, and what can we do about it?  Anders Sandberg explores one key issue:

If we want to increase the resiliency of our society we should work on increasing substitutability.  Devices and software should be able to use alternative infrastructures.  Knowledge of what can be substituted for what should be disseminated (so no time is lost when disaster strikes in trying to figure it out). This is particularly true in areas where many different kinds of inputs are needed.

Anders is right: our economy gets a bit more vulnerable to collapse each time a particular product or service uses a specialized input, available from only a few suppliers, rather than a general and widely available input.  For example, it is easier to replace a cell phone’s battery in a jam if that phone uses AA batteries, instead of a battery built especially for that phone.

Of course as the phone example illustrates, we wouldn’t want to arbitrarily require more general inputs to everything; much of our wealth comes from a division of labor, enabled by specialized inputs.  Regulatory agencies with discretion to declare which inputs must be general sound like a disaster.  Let us instead think like economists, and ask when market (or legal or state) failures may induce overly-specialized inputs:

  • Intellectual Property – many ideas for new products or services are not pursued because investors fear that if a first mover shows the product to be viable, it will be too easy for second movers to take over the market.  Investors prefer business plans centered on a specialized input that second movers cannot easily acquire, such as a key patent.  If we could create better incentives for innovating with general inputs, our economy would be less fragile.
  • Empire Bias – Firms are reluctant to buy specialized inputs, or to sell specialized products, or fear of being subject to extortion once two firms have become dependent on one another.  This fear makes firms either avoid specialized inputs, or merge such matched activities into a single firm.  Since manager empire-building already seems to make firms too large, ways to discourage this, such as enabling raiders, would also make our economy less fragile.
  • Crisis Metrics – Most contracts to buy inputs do not explicitly condition on collapse cues.  For example, the price we pay for electricity or phone service does not change when a collapse looms, even though we are willing to pay more to ensure continued service.  Such providers thus have insufficient incentives to ensure continued service in a crisis.  If we could publish independent metrics that flagged crisis situations, so that contracts could condition prices on such metrics, suppliers could have better incentives to avoid specialization that risked their reliability in a crisis.
  • Missing Standards – Firms often prefer incompatible standards in order to increase customer lock-in and reduce competition.  Policies that discourage incompatible standards would also reduce economic fragility.

While there are many ways to avoid specific disaster scenarios, the main general approaches I know are refuges, to directly protect against the worst case, and the robustness rewards above, which counter-act known problems that distort our world economy toward fragility.

GD Star Rating
Tagged as: , ,
Trackback URL:
  • Eliezer Yudkowsky

    If I build my business to be robust, and no one else does, then the people who produce my inputs will collapse anyway, and my investment will be for naught. So preventing economic collapse is a coordination problem.

    If everyone else builds their business to be robust, and I don’t, then the people who make my inputs are more likely to stay in business anyway. So there may be a free-rider, commons problem.

    Each party to a transaction captures only a portion of the gains generated from trade. When an automobile manufacturer goes out of business, they lose only the portion of the gains from trade that they were able to capture, but their customers lose additional economic value as well. One way of looking at this is that the velocity of trade is partially a public good – a rationale for government intervention in economic depressions that I’ve never heard advocated, not sure why.

    Another way of looking at it is that the benefits of resilience would be gained by both the manufacturer and the consumer, but the costs are borne by only one party at a time. Customers don’t spend much time thinking about and being willing to pay higher prices to manufacturers who invest in resilience to disaster, manufacturers don’t give discounts to customers who invest in unemployment insurance or large savings accounts that would enable them to go on consuming when the manufacturer needs more customers, and even if someone tried this there would be a large free-rider problem.

    Economic collapses are commons coordination problems – in fact, ordinary economic depressions are commons problems. This seems really obvious to me now, but it’s not something I can remember hearing pointed out. I probably just don’t read the right textbooks.

    • Norman


      I think you point out the real problem with economic crises. There is a lot of literature that touches on these issues, but not much that deals with it head on. It’s closely related to the literature on permanent recessions. Axel Leijonhufvud has some work touching on this issue, and you might look at Cerra and Saxena (AER, 2008). My dissertation (in progress) is in part an attempt to develop the intuition of this coordination problem as a link between short run recessions and long run stagnation. So hopefully there will be more work on the subject in the coming years.

  • anon

    I’m not sure that it makes sense to make our economy more resilient to collapse. Collapses are low-probability events, and even the most resilient economy will have a far lower productivity in the event of a collapse.

    So it may make more sense to just stockpile emergency supplies when times are good and they can be produced cheaply, and draw them down if a collapse occurs, without worrying too much about productivity during the crisis. If market mechanisms are preserved by the collapse and pricing is unrestricted, these stockpiles could even be accumulated by private entrepreneurs.

    Eliezer, in a perfectly competitive market it is the case that “each party to a transaction captures only a portion of the gains generated from trade”, but these gains are a pecuniary externality which does not impact efficiency.

    In the presence of fixed costs or sunk capital this is no longer the case, because goods must be priced far above marginal cost in order to recover these fixed costs. So a failure of “aggregate demand” can trigger a recession. Economists refer to these scenarios as “imperfect competition”, but this is a rather misleading term, since pricing above marginal cost (and price discrimination) can occur even in the absence of conventional ‘market power’.

  • Unimportant individual

    Well, one way to increase robustness and efficiency would be to just remove intellectual property. The incentive to remake existing technology would disappear(unless it’s actually better/cheaper) if third parties could make copies of proprietary devices at the production cost without breaking the law. And if one company stops making a particular product, or becomes too expensive, you could ask another company to replicate it for you.

    Sure, some inventors might feel cheated when they get ripped off, but it wouldn’t stop people and companies from continuing to invent. Just imagine it for a moment: a single charger for all of your electronic devices. Ahhh…

    Faux pas?

    • anon

      You can get many of the same effects simply by buying out the “intellectual property” rights after the fact and making the product freely reproducible. The reason this isn’t done is mostly because coordination costs are high and patents expire rather shortly anyway (and may be compulsorily licensed in case of urgency).

      Also, where the gains would be greatest, products tend to be rather free of IP restrictions anyway: think about how quickly open source software and open access publishing sprung up once the Internet raised the costs of IP restrictions.

  • Robin Hanson

    Eliezer, contracts with crisis-contingent prices give firms incentives to coordinate on crisis production plans, just as ordinary prices coordinate ordinary production plans. I won’t claim either coordination is perfect, and am open to suggestions to cheaply improve them.

    anon, we can have refuges/stockpiles and make the economy more robust.

    unimportant, if the loss of intellectual property slows growth, we may have a smaller economy. A smaller more robust economy may be worse overall, even regarding collapse.

    • Wei Dai

      Robin, it’s a high-status move for you to aggregate replies to multiple people on different threads into one comment. It might save you a few seconds of time, but causes a lot of inconvenience to your readers by breaking up the natural flow of conversations. Thought I’d point it out in case you’re still doing your status audit thing.

      • Eliezer Yudkowsky

        I don’t see this as a high-status move particularly but agree with the point about convenience.

      • Robin Hanson

        I am a big fan of ordinary natural human conversation, which is adapted in great detail to many voices speaking in sequence. I prefer blog comments to seem more like such a conversation. I don’t like the way nesting breaks up the conversation, and I distrust the move away from our inherited conversation style. I provide the nesting out of courtesy to those who comment, but I feel I should be allowed to converse my own way in my own blog comments. On other folks’ blogs, I’ll converse their way.

      • Zack M. Davis

        “I provide the nesting out of courtesy to those who comment, but”

        Arguably a consistent style is more important than threaded versus flat comments. Do you have the option of turning threading off on future posts?

      • komponisto

        Agree with Wei on it being a high-status move. Of course, RH should be entitled to assert high status here: it’s his own blog!

      • Wei Dai

        komponisto, Robin had previously expressed an interest in finding low cost ways to avoid high-status moves. See this post.

      • Michael Metcalf Bishop

        I would love it if Robin elaborates his explanation for replying to multiple people in one comment. Even if this is a pure-status-grab, I have a hard time complaining about someone making such moves on the fantastic free blog they write.

  • Prakash

    Nice post. Lots of disjoint thoughts on the same.

    Firms using JIT techniques are usually advised to keep large cash reserves to serve as a buffer. Instead of building a buffer in terms of goods, they build a reserve in cash.

    Robin, I think enabling raiders is a wrong strategy in this case. Companies with large buffers/reserves are easier targets for takeover. it is too easy to paint all those reserves being kept aside, as wastage, when it is actually resilience. You need someone with a long term view, which may not be corporate raiders. Most of the private equity universe seeks short term gains. (i consider even 3-5 year periods as short term, if we are talking about civilization wide resilience)

    The main problem remains what Eliezer has pointed out. Resilience acquires the nature of a commons problem/public good in the present system.

    Also, crisis contingent contracts might tempt people to play around the term “crisis”.
    Eg. Electricity is highly unreliable in India, a uniform pricing structure continues to be imposed. To counteract this, many households have electricity backups in terms of inverters with batteries. Now this can be painted as plain old inefficiency or it can be painted positively as resilience. I’m not sure how different would the situation be if the power went out, the electric company calls me and says, “this is a crisis. As per our contract, we can restore power if you pay us double.” i might agree once, and twice in dire circumstances, but after the third time, i will go for an inverter backup, bringing us back to situation one, where uniform pricing was there.

    Kindly explain if I have misinterpreted your suggestion.

    Another thought is – How far should one take Anders’s Substitutability suggestion? If you extend it from components to final consumer goods, you will end up with a Buddha who is satisfied with the most basic of things, because everything is substitutable for him/her.

    • Robin Hanson

      I talked about “independent” metrics exactly to avoid the moral hazard concerns you mention.

    • anon

      You need someone with a long term view, which may not be corporate raiders.

      In theory, this long term view should be provided by share traders and investors. But this doesn’t really happen because of asymmetric information. Is there a way to enable insiders to bet on the firm’s long-term profitability, while avoiding the issues which arise with ordinary insider trading?

      Also, crisis contingent contracts might tempt people to play around the term “crisis”… I’m not sure how different would the situation be if the power went out, the electric company calls me and says, “this is a crisis. As per our contract, we can restore power if you pay us double.”

      This can happen when electricity suppliers have market power: it probably did occur during California’s 2000 power crisis. But in the Californian case, the electricity distributor was actually forbidden to raise prices, so consumers did not conserve energy and higher-cost plants were not incented to come on line and supply more power. If the distributor had not been bound by price caps, that particular crisis would have been largely averted.

  • Robert Wiblin

    Find specific generic products which we will need in disasters and stockpile them (oil), or push up their prices so that we build up greater excess production capacity (food for example). If this is too command and control, we can get the same effect in the market by encouraging speculators to buy up and store these products by credibly committing to keep price gouging legal during crises.

  • Katja Grace


    “I am a big fan of ordinary natural human conversation, which is adapted in great detail to many voices speaking in sequence….”

    Perhaps the craziest sounding opinion I’ve heard you hold. What’s so good about it? Why is our move away from traditional conversation style untrustworthy?

  • Robin Hanson

    Rob, raising the crisis-price of goods that can be stockpiled should induce private stockpiles of such goods. When a crisis-price can differ from an ordinary price, consumers would naturally raise the crisis-price of goods they really need in a crisis.

    Katja, anything that is adapted in great detail to use in a particular environment may work less well in differing environments. Human conversation adaptations do well at not only helping us infer what others know, but also what they know that others know, what they know others know that still others know, and so on up the meta-knowledge hierarchy. Most nested comment systems do badly at supporting meta-knowledge.

    • Tyrrell McAllister

      Human conversation adaptations do well at not only helping us infer what others know, but also what they know that others know, what they know others know that still others know, and so on up the meta-knowledge hierarchy.

      Yes, the human brain is good at keeping track of the threads of a conversation. But it’s not infinitely good. Explicit threading is a technological assist that helps the brain keep track of threads even better.

      Analogously, human brains are good at keeping track of their associates, their family, friends, and acquaintances. But we still benefit from having address books.

  • botogol

    as a slight aside to the point of the article, but referencing to the actually example you gave, you might be interested to the know that the EU has mandated that all mobile phones must be rechargable via a USB connection.

    This is popular amongst users, especially with a family: 5 phones, one charger. Hurray. I would be interested to know whether manufacturers fought it or welcomed it.

  • Anders Sandberg

    One non-intuitive thing I noticed in my simulation of substitutes was that it didn’t matter noticeably if the possible substitutes for a good were randomly selected or had a power-law distribution (i.e. there are some goods that can substitute for a lot of things, while others are specific). Of course, this might be an artefact from not simulating whether demands could be met – generic products might suffer much stronger rise in demand than non-generic products in a crisis, and this could cause them to run out.

    Demanding open standards such as phone charging through USB or interoperability of document formats might be a way of forcing substitutability. I guess there is a sizeable economics literature on what conditions make industries want such standards? It might be worth looking at whether this could be generalized to other fields too – having more engines that can run on different kinds of fuel seems worthwhile.

    • Robin Hanson

      Yes inefficiently diverging standards also causes needless fragility; I’ll ad that to the list in the post.

  • Hal Finney

    Some people claim that a source of fragility is our systematic and institutional underestimation of the likelihood of serious collapse. The “black swan” critique by Nassim Nicholas Taleb runs along these lines. Perhaps one factor is the concern that bets in favor of collapse can’t win because there will be no infrastructure available to pay off if collapse happens. This could cause market prices, and decisions based on them, to systematically underestimate collapse risks.

    I wonder if we could do something with a small kernel of a market or similar forum which is considered very likely to survive collapse. Perhaps contracts could be denominated in gold, and the system be administrated by the Church. Would there be some way to leverage such a high-reliability subsystem and build more reliability into the larger economy?

    • anon

      Perhaps one factor is the concern that bets in favor of collapse can’t win because there will be no infrastructure available to pay off if collapse happens.

      This is almost certainly a factor. Robin has proposed bets involving refuge tickets to deal with the most severe cases, and some milder scenarios may be evaluated by proxies such as the price of gold and oil.

      Should we create a resilient infrastructure to make sure that bets about crisis scenarios are adjudicated? What would such an infrastructure look like?

    • mjgeddes

      You can start making money off prediction markets immediately, no matter how far into the future an event is, or what its consequences are. The probabilities shift up and down as more people put money on, and you simply ‘Back’ (bet for the event) at a high price, and ‘Lay’ (bet against the event) at a lower price. Simple trading, why can’t simple trades work even for long-term bets on an existential disaster?


      Initial Market on Unfriendly AI eating the world by 2020:

      For $1 investment
      Day 1 Market – Best Prices

      Back: $100 Lay: $112

      Ignorant punters have placed bets after Day 1 , I see that they’ve underestimated the odds and I ‘Back’ for big money.

      Day 2 Market – Best Prices

      Back: $20 Lay: $23

      A lot more money has gone on, some experts on AI have placed their bets, and the odds are now more reasonable. I lay.

      Result: Backed at $100, Layed at $23. Profit after only a day.

      • anon

        why can’t simple trades work even for long-term bets on an existential disaster?

        Because prediction markets need to be adjudicated after the event occurs, otherwise they don’t work. Unless the contracts happen to be denominated in paperclips, it’s quite difficult to settle them after an unfriendly AI has taken over the world.

      • mjgeddes

        But most trades are settled before the event occurs. I routinely trade on Betfair and the profit/loss is credited to my account right away – I don’t have to wait for the event to occur.

  • Pingback: Tweets that mention Overcoming Bias : Reward Robustness --

  • Anders Sandberg

    Just noticed a correspondance to Nature (, Could a boom in technologies trap Feynman’s simulator? by Dietrich Leibfried) that suggests another reason to get substitutability: the need for certain robust technologies even without the disaster. In the letter Leibfried notes that we may need several different technological implementations of quantum physics simulators to make sure they are giving accurate results. In the case of internet standards, there has been the tradition of demanding at least two different implementations of the same proposed standard, so that there is good reason to think it works.

    I’m not certain this technological demand is general enough to affect all of society, but maybe one could tweak standardization body rules and public call for bids to favour technologies with multiple implementations.

  • Pingback: Making the Human Brain Stronger; A Theory

  • Pingback: Overcoming Bias : Space vs. Time Allies