I’ve often tried to help [convince] companies do experiments, and usually I fail spectacularly. … Companies pay amazing amounts of money to get answers from consultants with overdeveloped confidence in their own intuition. Managers rely on focus groups — a dozen people riffing on something they know little about — to set strategies. And yet, companies won’t experiment to find evidence of the right way forward. I think this irrational behavior stems from two sources. One, … experiments require short-term losses for long-term gains. … Second, there’s the false sense of security that heeding experts provides.
What about governments? Why don't they experiment? When they pass a trillion-dollar bailout or stimulus bill, do they even spend a billion dollars to study whether it worked or not?
We have some states that oppose a bill, and some that want it. That's perfect! Instead of arguing over what every state in the union should do, we should argue over how to divide the states up into states that will implement the new law, and states that won't, so that the experiment will be as well-controlled as possible.
I think that sociology as well as economics plays an important role in understanding firm behavior. There's a strong stereotype of corporate BS, and there must be reasons for this. The real question for someone like me is whether it's possible to avoid this sociological denigration. From a Paul Graham essay:
At one point [Mitch Kapor and the head of HR at Lotus] worried Lotus was losing its startup edge and turning into a big company. So as an experiment she sent their recruiters the resumes of the first 40 employees, with identifying details changed. These were the people who had made Lotus into the star it was. Not one got an interview.
It seems likely that playing much closer attention to hiring decisions than usual might help a firm avoid sociological denigration, but this would probably also make it more difficult to grow.
I suspect that the expectation of security from following consultant's advice is quite real. A modest degree of principal-agent problems allows a manager to take credit for success and blame consultants for failure.
Cody L. Custis, you seem to be referring to companies taking risks. But a company could also do an experiment just to gather data before taking such risks, and that is the question being asked here.
Simon K., your distinction between different domains of experimentation seems useful. But asking "why don't economists do X more" doesn't seem quite analogous. Do economists display a reluctance to conduct such experiments when someone offers the idea? Do they spend significant amounts on consultants or focus groups instead?
Given that most people outside of firms as well as other organizations prefer impressive consultants to experiments, shouldn't a better title be "people won't experiment"? Or do firms experiment less than other entities?
From where I'm sitting it looks like firms experiment more than other organizations. Of course they still may not do enough experiments.
Engineering firms do experiments constantly. Otherwise nothing we did would ever work. I think the question is really "why don't firms experiment on their customers and staff"? But this is a question analogous to "why don't economists rely more on experiments"? The answers are the same - the obvious experiments are prohibitively expensive and damaging, the data obtained from "natural experiments" is unreliably and hard to interprety, and small scale feasible experiments often produce results that aren't replicated on a larger scale (New Coke, anyone?). The effects are probably somewhat larger in management that in economics - firms care more about their reputations than economics professors (or even politicians), and data sets tend to be small and confidential to specific firms or even business units.
As a management consulting for one of the top three, let me point out two I would think to be pretty obvious things:1) Many firms would not know how to setup experiments even if they wanted. It's not just a mindset but also a talent thing. The longer I do this, the more I become convinced that talent is perhaps the scarcest resource we have2) Many questions addressed to consultants are of the bet your company type. You simply cannot try and experiment, because the brick wall is approaching. Fast.
As for Joe, you are absolutely right, you cannot sell an idea to most companies with lets try - without clear management backing and according incentives, many people will do well whatever. There are few companies -mostly smaller, start up kind of ones and a few of the big research heavy juggernauts- who still do it, but even then, it's not like "let's try this" but more of "this seems to be in line with strategy, let's see where we are with it in X, if we fail to achieve Z we will kill it".
Most web companies (like Amazon and Google) do hundreds of experiments per year, mostly in the form of "split testing" (or "a/b tests"). I'm sure that large brick-and-mortar retailers like Wal-Mart do the same.
i would just like point out that you are using the fact that one economics professor is anecdotally saying that firms don't experiment to say that firms don't experiment.
just because ariely has found that a few companies he has worked with might not do experiments doesn't mean that all or even most companies don't do experiments...maybe he affiliates with the wrong companies
now believe me, i do agree with what he is saying
working in a big firm, my intuition is that in general firms don't like to experiment...but how can you use the intuition of a prestigious person you are affiliating with to point out the folly of firms doing the very same thing?
Wait. If private firms owned by self-interested shareholders won't invest in a method that's supported by self-interested betting... surely there's an element of contradiction here? Why are we assuming that shareholders are failing to be self-interested but the predictors will not be? Are principle-agent problems this large?
And, well, prediction markets are driven by people who think they know the future. Firms are run by people who think they know the future. Ultimately someone has to be doing the predicting, and if they are, why not just buy into the relevant firms directly rather than by betting in a prediction market? The returns there can hardly be greater.
We should observe relatively little experimentation within firms if markets really do punish mistakes quickly. If they don't, then... well, depending on the type of market failure here, prediction markets might not do so well either.
I think there is more to this than "firms won't experiment". I studied chemical engineering in school (recently graduated) and worked at two different paper manufacturing firms as an intern. My projects at both firms involved conducting and evaluating experiments, and I observed others do several of their own (expensive to run) experiments. These were well organized and relatively formal experiments.
Many school projects designed to look like "real world projects" were also formulated as experiments done within a firm, so I suspect experiments are widespread.
Perhaps firms that are not used to conducting experiments are reluctant to start? Or perhaps people are reluctant to do experiments on people?
It's likely that managers don't so much have a problem with experimenting as with doing so openly. Not (only) because of the message this sends to outsiders that they don't really know the right path ahead, but more importantly because it opens everything they hand down the chain of command to question. Companies I've worked for have tried internal initiatives that didn't work (e.g. incentive schemes, projects, employee regulations) and then tried alternatives, but each alternative was presented with a semi-plausible story about positive goals to sell the new initiative. I'm sure managers often say to one another things like "well let's try this out for 6 months and see how it goes", but once the decision is made it is invariably presented to everyone else as a strong ongoing commitment. I suppose that at the highest-profile levels (C*Os and boards of large public companies) it becomes more difficult to conspire to experiment.
What about governments? Why don't they experiment? When they pass a trillion-dollar bailout or stimulus bill, do they even spend a billion dollars to study whether it worked or not?
We have some states that oppose a bill, and some that want it. That's perfect! Instead of arguing over what every state in the union should do, we should argue over how to divide the states up into states that will implement the new law, and states that won't, so that the experiment will be as well-controlled as possible.
I think that sociology as well as economics plays an important role in understanding firm behavior. There's a strong stereotype of corporate BS, and there must be reasons for this. The real question for someone like me is whether it's possible to avoid this sociological denigration. From a Paul Graham essay:
At one point [Mitch Kapor and the head of HR at Lotus] worried Lotus was losing its startup edge and turning into a big company. So as an experiment she sent their recruiters the resumes of the first 40 employees, with identifying details changed. These were the people who had made Lotus into the star it was. Not one got an interview.
It seems likely that playing much closer attention to hiring decisions than usual might help a firm avoid sociological denigration, but this would probably also make it more difficult to grow.
I suspect that the expectation of security from following consultant's advice is quite real. A modest degree of principal-agent problems allows a manager to take credit for success and blame consultants for failure.
There's a reputational risk if a firm is observed to offer different prices for the same good. This happened to Victoria&aposs Secret.
Cody L. Custis, you seem to be referring to companies taking risks. But a company could also do an experiment just to gather data before taking such risks, and that is the question being asked here.
Simon K., your distinction between different domains of experimentation seems useful. But asking "why don't economists do X more" doesn't seem quite analogous. Do economists display a reluctance to conduct such experiments when someone offers the idea? Do they spend significant amounts on consultants or focus groups instead?
Given that most people outside of firms as well as other organizations prefer impressive consultants to experiments, shouldn't a better title be "people won't experiment"? Or do firms experiment less than other entities?
From where I'm sitting it looks like firms experiment more than other organizations. Of course they still may not do enough experiments.
Engineering firms do experiments constantly. Otherwise nothing we did would ever work. I think the question is really "why don't firms experiment on their customers and staff"? But this is a question analogous to "why don't economists rely more on experiments"? The answers are the same - the obvious experiments are prohibitively expensive and damaging, the data obtained from "natural experiments" is unreliably and hard to interprety, and small scale feasible experiments often produce results that aren't replicated on a larger scale (New Coke, anyone?). The effects are probably somewhat larger in management that in economics - firms care more about their reputations than economics professors (or even politicians), and data sets tend to be small and confidential to specific firms or even business units.
That should obviously read consultant. :)
As a management consulting for one of the top three, let me point out two I would think to be pretty obvious things:1) Many firms would not know how to setup experiments even if they wanted. It's not just a mindset but also a talent thing. The longer I do this, the more I become convinced that talent is perhaps the scarcest resource we have2) Many questions addressed to consultants are of the bet your company type. You simply cannot try and experiment, because the brick wall is approaching. Fast.
As for Joe, you are absolutely right, you cannot sell an idea to most companies with lets try - without clear management backing and according incentives, many people will do well whatever. There are few companies -mostly smaller, start up kind of ones and a few of the big research heavy juggernauts- who still do it, but even then, it's not like "let's try this" but more of "this seems to be in line with strategy, let's see where we are with it in X, if we fail to achieve Z we will kill it".
Most web companies (like Amazon and Google) do hundreds of experiments per year, mostly in the form of "split testing" (or "a/b tests"). I'm sure that large brick-and-mortar retailers like Wal-Mart do the same.
i would just like point out that you are using the fact that one economics professor is anecdotally saying that firms don't experiment to say that firms don't experiment.
just because ariely has found that a few companies he has worked with might not do experiments doesn't mean that all or even most companies don't do experiments...maybe he affiliates with the wrong companies
now believe me, i do agree with what he is saying
working in a big firm, my intuition is that in general firms don't like to experiment...but how can you use the intuition of a prestigious person you are affiliating with to point out the folly of firms doing the very same thing?
Wait. If private firms owned by self-interested shareholders won't invest in a method that's supported by self-interested betting... surely there's an element of contradiction here? Why are we assuming that shareholders are failing to be self-interested but the predictors will not be? Are principle-agent problems this large?
And, well, prediction markets are driven by people who think they know the future. Firms are run by people who think they know the future. Ultimately someone has to be doing the predicting, and if they are, why not just buy into the relevant firms directly rather than by betting in a prediction market? The returns there can hardly be greater.
We should observe relatively little experimentation within firms if markets really do punish mistakes quickly. If they don't, then... well, depending on the type of market failure here, prediction markets might not do so well either.
And the consultants aren't interested in using prediction markets since they are valued for confidence not accuracy?
I think there is more to this than "firms won't experiment". I studied chemical engineering in school (recently graduated) and worked at two different paper manufacturing firms as an intern. My projects at both firms involved conducting and evaluating experiments, and I observed others do several of their own (expensive to run) experiments. These were well organized and relatively formal experiments.
Many school projects designed to look like "real world projects" were also formulated as experiments done within a firm, so I suspect experiments are widespread.
Perhaps firms that are not used to conducting experiments are reluctant to start? Or perhaps people are reluctant to do experiments on people?
It's likely that managers don't so much have a problem with experimenting as with doing so openly. Not (only) because of the message this sends to outsiders that they don't really know the right path ahead, but more importantly because it opens everything they hand down the chain of command to question. Companies I've worked for have tried internal initiatives that didn't work (e.g. incentive schemes, projects, employee regulations) and then tried alternatives, but each alternative was presented with a semi-plausible story about positive goals to sell the new initiative. I'm sure managers often say to one another things like "well let's try this out for 6 months and see how it goes", but once the decision is made it is invariably presented to everyone else as a strong ongoing commitment. I suppose that at the highest-profile levels (C*Os and boards of large public companies) it becomes more difficult to conspire to experiment.