Manzi On Trials, Consulting

Arnold Kling on Jim Manzi’s new book Uncontrolled:

Manzi is a fan of randomized controlled experiments in business and public policy (in the latter, examples include the Rand health care study and the Wisconsin income-maintenance studies). I believe that decision-makers will resist this approach, for the same reason that they resist Robin Hanson’s suggestion to use prediction markets. That is, decisions are not necessarily about achieving results. They are often about establishing the status of the decision-maker. For a decision-maker to conduct experiments or to employ prediction markets is to admit ignorance and doubt, which lowers the decision-maker’s status.

Manzi responds:

I agree that this is true, and is a big deal. In the book, I expend a fair amount of effort describing the procedures and methods that have been used to ameliorate this problem (though never eliminate it) in therapeutic medicine, many large businesses, and certain narrow areas of government policy development. I think at a more strategic level, however, this problem is best addressed by decentralizing authority and accountability. Staff businesspeople, academics, and so on have much larger incentives to use “analysis as rhetoric” in the manner that Kling refers to than do people who are responsible for achieving outcomes in a marketplace. If I am paid (or live or die) based on my programs working or not, I am much more likely to care about what really works rather than getting tangled up in what analysis will get me noticed and promoted.

The book isn’t out yet. Kling got an advanced copy, but I did not. I look forward to seeing Manzi’s detailed discussion, but the above response seems to miss the point – authority and accountability won’t be decentralized if that lowers the status of central folks. Just because they should decentralize doesn’t mean they will.

Similarly, a few weeks ago Manzi responded to my post on the puzzles of why firms pay so much for often trite consulting advice, and why such advisors hire so many fresh grads of top schools. I suggested that firms are more buying prestige to bully locals into cooperation than they are buying info per se, and that recent top school grads offer the most prestige per wage dollar. Manzi disagreed:

A typical team might include three research analysts with 0–2 years of experience each, two associates with 3–5 years of business experience each, an engagement manager with 5–7 years of experience, and a junior partner with about ten years of experience. Such a team is ultimately supervised by a senior partner who has overall responsibility for the relationship between the firm and the client, reviews the outputs, and attends key client meetings, but does not directly lead the specific project. No competent consulting firm is going to have a bunch of unsupervised “kids fresh out of college” standing in front of a Fortune 500 CEO telling him what they think. …

Not all projects add value … but a typical set of recommendations might be something like raising the prices for the following list of products, based on a combination of: (1) historical price elasticity analysis … (2) activity-based costing … and (3) competitive analysis to estimate the likely competitor price response. …

The CEO often understands what needs to be done, but does not have the resources to fight this blocking coalition. But if a prestigious outside consulting firm weighs in, that can turn the status tide.

It’s often the case that a CEO wants to institute a new CRM system for the sales force. … But having McKinsey write a report on the value of the CRM system is unlikely to make the head of sales yield because she just can’t “resist the further support of a prestigious outside consultant.” If anything, when a CEO uses a report, in part, to get a stamp of approval, it’s more likely to be for upward management of the board.

The board counts as locals in need of bullying, in my view. And Manzi seems to again miss the point – his story here doesn’t even try to explain the two key puzzles of 1) why firms are willing to pay such huge sums for analyses of price elasticity, costing, and competitor responses that they could seemingly do themselves cheaper, and 2) why consulting firms seem so much more eager than other firms to pay steep premiums to hire fresh top school grads, when lesser school grads seem nearly as capable and much cheaper, and more experienced folks would seem more useful.