Discover more from Overcoming Bias
I’ve often tried to help [convince] companies do experiments, and usually I fail spectacularly. … Companies pay amazing amounts of money to get answers from consultants with overdeveloped confidence in their own intuition. Managers rely on focus groups — a dozen people riffing on something they know little about — to set strategies. And yet, companies won’t experiment to find evidence of the right way forward. I think this irrational behavior stems from two sources. One, … experiments require short-term losses for long-term gains. … Second, there’s the false sense of security that heeding experts provides.
More here (HT Tyler). Wow – and I was puzzled by firms disinterest in prediction markets; experiments have a much better intellectual pedigree. Hiring consultants allows one to affiliate with prestigious folks, while focus groups allow one to brag about “listening to the people” (which is in fact how prediction markets are usually sold). Apparently actually improving decision quality is way way down in the manager priority list.
Firms Won’t Experiment
What about governments? Why don't they experiment? When they pass a trillion-dollar bailout or stimulus bill, do they even spend a billion dollars to study whether it worked or not?
We have some states that oppose a bill, and some that want it. That's perfect! Instead of arguing over what every state in the union should do, we should argue over how to divide the states up into states that will implement the new law, and states that won't, so that the experiment will be as well-controlled as possible.
I think that sociology as well as economics plays an important role in understanding firm behavior. There's a strong stereotype of corporate BS, and there must be reasons for this. The real question for someone like me is whether it's possible to avoid this sociological denigration. From a Paul Graham essay:
At one point [Mitch Kapor and the head of HR at Lotus] worried Lotus was losing its startup edge and turning into a big company. So as an experiment she sent their recruiters the resumes of the first 40 employees, with identifying details changed. These were the people who had made Lotus into the star it was. Not one got an interview.
It seems likely that playing much closer attention to hiring decisions than usual might help a firm avoid sociological denigration, but this would probably also make it more difficult to grow.