Fail Faster

It looks bad for a manager to have one of his projects fail. So to “cover his ass”, such a manager often tries to prevent any records showing that people saw failure coming. After a failure, he wants to say “this was just random bad luck; no one could have foreseen seen it.” His bosses up the chain of command tend to allow this, because they also want to avoid being held responsible for failures during their watch. So they also prefer the random back luck story.

Unfortunately, this approach tends to prevent organizations from getting signals that would let them mitigate failures, such as by quitting projects earlier. For example, most startup firms don’t fail until they have spent nearly all of the cash they were given. It is rare for a startup to admit it isn’t going to work out, and give some cash back to investors. Similarly, government agencies created to achieve some purpose rarely recommend to legislatures that they be eliminated when their find that they aren’t achieving their intended purposes.

Of course bosses don’t want to be too obvious about silencing possible signals of failure. They find it hard to silence what have become standard signals, like cost accounting measures.

A great application of prediction markets is to give better and clearer warnings of upcoming failure, to enable better mitigation, such as quitting. Of course project bosses anticipate this, and oppose prediction markets on their projects, for exactly this reason. But we can still hope that prediction market warnings may someday become a standard signal, and thus hard to silence:

I hope prediction markets within firms may someday gain a status like cost accounting today. In a world were no one else did cost accounting, proposing that your firm do it would basically suggest that someone was stealing there. Which would look bad. But in a world where everyone else does cost accounting, suggesting that your firm not do it would suggest that you want to steal from it. Which also looks bad.

Similarly, in a world where few other firms use prediction markets, suggesting that your firm use them on your project suggests that your project has an unusual problem in getting people to tell the truth about it via the usual channels. Which looks bad. But in a world where most firms use prediction markets on most projects, suggesting that your project not use prediction markets would suggest you want to hide something. (more)

GD Star Rating
loading...
Tagged as: ,
Trackback URL:
  • IMASBA

    “After a failure, he wants to say “this was just random bad luck; no one could have foreseen seen it.”

    Yeah, it’s funny how that never works the other way around: they’ll never say “this huge succes” was just random luck. No one is worth $10 million a year.

    Prediction markets can provide the role of keeping records of what other people thought the chances of the project were before it became a hit or a miss. There are three problems though: 1) such a prediction market would be vulnerable to manipulations by people who benefit from making the chances of the project appear better or worse than they actually are, 2) even if the prediction market is right about the expected value there still is variance, so the prediction markets will still be “wrong” plenty of times, human psychology being what it is means people will then distrust prediction markets, even if a statistical analysis shows the expected value predicted by the market was right, 3) these humans kinda have a point without realizing it: often it matters what the expected value of succes is times the expected value of the reward of succes, so consistently avoiding projects that are likely to fail might not be beneficial for mankind in the long run (of course you can have a prediction market that takes this into account, but it’s only going to convince investors if the number of events per unit of time is high, which is often not true in the real world).

    • http://overcomingbias.com RobinHanson

      Most every time I mention prediction markets here you mention manipulation, even though I’ve repeatedly pointed you toward the literature on that subject. Read and respond to the literature or … shut up.

      • IMASBA

        I’ve already responded that the literature usually only takes into account the market itself, not the complete system of market plus whatever it is the market aims to predict. Investing $1 million to manipulate a prediction market enough to yield you a $1 billion contract is money well spent even if you end up losing that $1 million (specifically I used the fictional example of a prediction market on the environmental effects of the Keystone XL pipeline where that market is highly unlikely to be in the billions of dollars range, even though the profit from the pipeline will be) I have yet to see a response to this and similar scenarios (I expect it to be very common that the prediction market is smaller than whatever deal or activity depends on the index of that prediction market.)

      • http://overcomingbias.com RobinHanson

        The literature deals with that sort of issue at great length. Showing that you have not in fact read the literature.

      • IMASBA

        I’ve read a lot of material on prediction markets, though admittedly I’m of course less of an expert than you are and so may have missed some important paper.

        All I keep finding on the problem I posed is that proponents say more traders will be attracted to the market to make a profit by betting against manipulators, you’re addition to this seems to be that you existing traders in the market will take another look at their bets, putting more effort into getting information, and then bet more heavily against the manipulators, that’s clever, but it a) depends on traders knowing there are significant attempts at manipulation and b) deep pockets of the non-speculative traders, for exampe they may have to spend a large multiple of what the manipulators are spending to restore the market price to something resembling the certainty of a scientific advisory report (additional problem: since extreme prices hint at manipulation then traders may be fooled into believing a genuine scientifically backed position is an attempt at manipulation). Even when the traders don’t have to outspend the manipulators by much the market can still quickly run into real world limitations since income and wealth inequalities in the world are truly staggering (and foreign governments can literally print money if they run out). In of the papers you yourself admit real world limitations of the pocket depth of traders can be an issue and even limiting the amount of money a single trader can bet leaves the possibility of a conspiracy (conspiracies aren’t very likely, what is likely is the Koch brothers setting up a multitude of paper companies each with their own bank accounts through which betting money can be channeled, or them simply giving their employees $1000 each and telling them they can bet on the Koch’s desired position on the prediction market, or get their asses fired).

        So I reiterate, who’s going to do something about the Koch brothers and their ilk pouring billions into prediction markets, what solution do you offer? Does Google have enough money to crush upcoming promising competitors by manipulating prediction markets on the future succes of those competitors.

        The UK is the perfect test ground for prediction markets (almost no gambling regulations and a public eager to place bets). Any large scale tests with open prediction markets should take place there and then we can see how prediction markets fair against expert predictions.

      • Zingram

        You have no idea how refreshing that was to read, coming from you. Props.

  • brendan_r

    From a stock picking perspective, the easiest insights are of the negative variety. Good news flows and is priced fast because it’s publicized. It’s easier to add value by seeing the bad before others do; projects that should be abandoned, geographic expansions, etc. And since stock picker’s have much worse info than insiders, there’s gotta be easy fruit for an insider’s prediction market to pick.

  • http://www.gwern.net/ gwern

    > For example, most startup firms don’t fail until they have spent nearly all
    of the cash they were given. It is rare for a startup to admit it isn’t
    going to work out, and give some cash back to investors.

    Not sure if investors really want startup firms to claim “it isn’t going to work out”… The Startup Genome project says that most successful startups wind up ‘pivoting’ from their original plan, which is startupese for ‘fail at original plan & come up with something else’. Even if they have no better odds after pivoting than a fresh young startup, they’ve at least paid all the fixed costs of getting started.

    • http://overcomingbias.com RobinHanson

      Whatever new plan they pivot toward, it will likely fail. And it could be better for investors to fail earlier, without spending all the cash.

      • http://www.gwern.net/ gwern

        > Whatever new plan they pivot toward, it will likely fail.

        So? Startups usually fail, whether it’s the first plan or the second plan.

        > And it could be better for investors to fail earlier, without spending all the cash.

        That’s not a response, that’s a restatement of your original claim I am criticizing. I gave two reasons why it’s better for investors for them to not give up at the first pivot.

      • http://overcomingbias.com RobinHanson

        I’m saying maybe they should give up instead of making a second pivot.

      • http://www.gwern.net/ gwern

        Yes, I believe that’s the third time you’ve said or implied that now.

      • Brent Dill

        From a purely resource-allocation perspective:

        Let’s say I’m the CEO of a startup, with a regular working relationship with my employees, and a contractual obligation to repay my investors. My employees are people that I interact with, day in and day out. I know their lives and their needs.

        I see that my venture might turn out unsuccessful. I have a choice: I can transfer the remaining money to my employees, by continuing to pay their salaries and “hope for a turn-around” while letting them know that things are going south, or I can close up shop, dump them all on the street, and repay my distant investors.

        As a human being, which am I more likely to want to do? Therefore, which am I more likely to find a rationalization for?

        Further, those investors took a substantial known risk betting on my venture. The employees just needed a place to work. Who is better equipped to survive not getting their money? Who is more likely to *expect* to not get paid?

        (From an economics perspective, this is terrible. But we aren’t always homo economicus).

      • http://juridicalcoherence.blogspot.com/ Stephen Diamond

        I gave two reasons why it’s better for investors for them to not give up at the first pivot.

        Always? Surely it’s use to obtain information discriminating projects by likely success. (Whether it’s worst its cost is always the question.) Your point seems merely obstinate.

      • http://www.gwern.net/ gwern

        > Always? Surely it’s use to obtain information discriminating projects by likely success. (Whether it’s worst its cost is always the question.)

        ‘Always?’ Probably not, but we are discussing startups here, an unusual and small area of the economy, and my previous points were startup-specific. ‘Surely it’s use’…? I don’t know what you mean here. I’ll assume you mean something like Brent Dill’s point, ‘surely startups should decide to close after one failure, and only refuse due to principal-agent problems'; I disagree for the previous reasons, and I’ll add some more: why are you privileging your own perspective on the matter, and ignoring the obvious facts that many investors (like Paul Graham and other VCs) continue to invest in startup companies? And further, are not routinely using their leverage to break up or dissolve startups the instant it decides to pivot?

        > Your point seems merely obstinate.

        It’s not obstinate to point out the repeated failure to reply to one’s points.

    • Nathan Merrill

      My dad actually recently received a check back from a start-up he put money into because they realized it wasn’t going to work out.

      Kind of crazy when people are responsible like that.

  • http://igmus.org/ Jeremy Wohl

    I’m curious if this line is not in fact ironic and you have some examples: “government agencies … rarely recommend … they be eliminated”. ?

  • Big biz

    Great post, Robin. On a tangent, when a company’s stock price tanks on the secondary markets (reflecting diminished prospects for that business going forward), managers often use the “public is too focused on quarterly earnings” excuse, and it’s the money managers that talk about bad luck (“everyone is down”).

    • brendan_r

      “…managers often use the “public is too focused on quarterly earnings” excuse…”

      ha yup, and yet, the persistence of downside momentum/drift/serial-correlation is much greater than rising. Investors, if anything, pay too little attention to bad news.

      Among financials, provisions for loan losses drift, too. Rising provisions predict more rises, though if accounting rules were followed, q to q provisions should be independent.

      Bad info flows slowly everywhere.

      • Doug

        ” Investors, if anything, pay too little attention to bad news.”

        I agree with the general sentiment of what you’re saying, but I think the picture is more complex. Depending how/when you measure it stocks outperform on earnings release dates by 0.1-0.5%. This suggests that most times investors are biased to expecting unknown information to be worse than it actually is. A similar phenomenon occurs across the broader market on Fed announcement days.

        But in general you are right, quant finance tells us that companies that have missed their most recent earnings strongly underperform. Managers want to convey the sense of long-term value building. But for every Steve Jobs there’s 100 mediocre CEOs. Probabilistically a company missing its earnings target is much more likely to be due to incompetence than long-term strategy.

      • IMASBA

        “Probabilistically a company missing its earnings target is much more likely to be due to incompetence than long-term strategy.”

        Right, but with that attitude mankind would be doomed. It doesn’t matter that 99% of the times it’s incompetence, it matters that that 1% pays off so much that the world would be a worse place if we didn’t let people businesses try to be among that 1%.

      • Don Zcar

        I agree. If failures were not tolerated then the failures of Apple Newton etc would never have lead to the successful IPHONE. If we don’t encourage trying and punish failures then we will never get anywhere.

  • Pingback: Recomendaciones | intelib

  • Pingback: Recommended Readings Week 1/3/14 | Pablo Martínez Bernal

  • josh

    My favorite is the boss who would rather fail according to plan, so as to have good metrics to show data on a failure that failed as per usual; rather that than to have an underling engineer a better solution which might save the program.

  • J Fritz

    I wonder how that would play out with those small bands of Ukrainian soldiers still holed up in isolated bases in Russia-controlled Crimea, refusing to surrender? Pretty much 100% chance of defeat. Should they just give up?

  • Daublin

    This particular bias might be beneficial on its own. An individual is more likely to make an idea succeed if they imagine it is plausible and then single-mindedly try to make it real. Thus, we get better results as a group if individuals are sometimes wildly biased.

    Courts take advantage of this bias, too. Each disputant has an advocate making their best case possible. The disputants are biased, but the overall system makes a better decision than if a circle of elders were to have a congenial chat about it.

  • col guy

    if you have a good job at a startup (in boston, today, in my field, biotech, that means salary >150,000 per year) why would you jepordize it by shutting down the company ?
    the longer you stay where you are, the better off you are, on avg; the ultimate of this is the post ipo zomby, where you take the 20,40, million from the ipo and coast till retirement

    i mean seriously, think about it from the day to day paycheck perspective of the worker

  • perfectinvesting

    Investments To Make Money – 250% daily for 20 days
    InvestmentsToMakeMoney.com Investment To Make Money Network has been established by United State of America’s InvestmentsToMakeMoney.com companies. After gaining experience in Forex trading for many years, these companies decided to form an investment firms besides their trading in order to accept investment funds from members and deposit for Forex trading. InvestmentsToMakeMoney.com Investment Network has been legally registered as a private investment firm.
    http://www.investmentstomakemoney.com