Manzi On Trials, Consulting

Arnold Kling on Jim Manzi’s new book Uncontrolled:

Manzi is a fan of randomized controlled experiments in business and public policy (in the latter, examples include the Rand health care study and the Wisconsin income-maintenance studies). I believe that decision-makers will resist this approach, for the same reason that they resist Robin Hanson’s suggestion to use prediction markets. That is, decisions are not necessarily about achieving results. They are often about establishing the status of the decision-maker. For a decision-maker to conduct experiments or to employ prediction markets is to admit ignorance and doubt, which lowers the decision-maker’s status.

Manzi responds:

I agree that this is true, and is a big deal. In the book, I expend a fair amount of effort describing the procedures and methods that have been used to ameliorate this problem (though never eliminate it) in therapeutic medicine, many large businesses, and certain narrow areas of government policy development. I think at a more strategic level, however, this problem is best addressed by decentralizing authority and accountability. Staff businesspeople, academics, and so on have much larger incentives to use “analysis as rhetoric” in the manner that Kling refers to than do people who are responsible for achieving outcomes in a marketplace. If I am paid (or live or die) based on my programs working or not, I am much more likely to care about what really works rather than getting tangled up in what analysis will get me noticed and promoted.

The book isn’t out yet. Kling got an advanced copy, but I did not. I look forward to seeing Manzi’s detailed discussion, but the above response seems to miss the point – authority and accountability won’t be decentralized if that lowers the status of central folks. Just because they should decentralize doesn’t mean they will.

Similarly, a few weeks ago Manzi responded to my post on the puzzles of why firms pay so much for often trite consulting advice, and why such advisors hire so many fresh grads of top schools. I suggested that firms are more buying prestige to bully locals into cooperation than they are buying info per se, and that recent top school grads offer the most prestige per wage dollar. Manzi disagreed:

A typical team might include three research analysts with 0–2 years of experience each, two associates with 3–5 years of business experience each, an engagement manager with 5–7 years of experience, and a junior partner with about ten years of experience. Such a team is ultimately supervised by a senior partner who has overall responsibility for the relationship between the firm and the client, reviews the outputs, and attends key client meetings, but does not directly lead the specific project. No competent consulting firm is going to have a bunch of unsupervised “kids fresh out of college” standing in front of a Fortune 500 CEO telling him what they think. …

Not all projects add value … but a typical set of recommendations might be something like raising the prices for the following list of products, based on a combination of: (1) historical price elasticity analysis … (2) activity-based costing … and (3) competitive analysis to estimate the likely competitor price response. …

The CEO often understands what needs to be done, but does not have the resources to fight this blocking coalition. But if a prestigious outside consulting firm weighs in, that can turn the status tide.

It’s often the case that a CEO wants to institute a new CRM system for the sales force. … But having McKinsey write a report on the value of the CRM system is unlikely to make the head of sales yield because she just can’t “resist the further support of a prestigious outside consultant.” If anything, when a CEO uses a report, in part, to get a stamp of approval, it’s more likely to be for upward management of the board.

The board counts as locals in need of bullying, in my view. And Manzi seems to again miss the point – his story here doesn’t even try to explain the two key puzzles of 1) why firms are willing to pay such huge sums for analyses of price elasticity, costing, and competitor responses that they could seemingly do themselves cheaper, and 2) why consulting firms seem so much more eager than other firms to pay steep premiums to hire fresh top school grads, when lesser school grads seem nearly as capable and much cheaper, and more experienced folks would seem more useful.

GD Star Rating
Tagged as: ,
Trackback URL:
  • Robert Koslover

    From my anecdotal experiences, I believe that the importance of status, or alternatively, the problem-solving methods that tend to be linked status, appear to vary by the type of organization. Those who misunderstand those factors in any particular organization risk having their status reduced. Real story: In my early years our of school, I worked for organizations that, for the most part, sought genuine technical answers to tough technical questions. When asked my opinion, it was acceptable (and even appreciated) for me to answer “I don’t know,” if, in fact, I didn’t know. So I behaved similarly when I later changed jobs to an organization that was (mostly) dedicated to manufacturing of commercial-world devices. I soon found that telling management “I don’t know” in answer to a technical question was heavily frowned-upon (to say the least). I eventually quit and went to work for a more research-oriented organization, where, once again, the answer “I don’t know,” if and when I don’t know, is entirely acceptable.

  • Robin,

    From my work in IT as a trainer, evaluator, and internal resource to-be-used-as-a-foil-for-consultants….

    1. Experience massively trumps education for recommendation usefulness. Folks from Harvard with 1 year of experience are worthless (worth less than zero for the quality of their recommendations) IT consultants…and folks from Podunk State with 15 years of experience are very good (on average)….IF what you are buying is correct approaches.

    2. Rarely do you use consultants to learn what to do. Rather, most of the time, consultants are used as a bludgeon for one group to get their way, by having the consultants agree with them. Good consultants (myself included at times), know that the job is to agree with the person paying, not to provide useful information.

    3. Occasionally you use consultants to do work that you don’t know how to do…or don’t have enough people to know how to do. Very often, they don’t know how to do it either…even though they don’t admit that to you. The usefulness of the generalist consultant from Harvard is that on average, their IQ/learning speed is higher, and can come up to speed faster on a new technology that they’ve never seen until yesterday. Of course, someone with 3 years experience with that technology is far more useful…but unless you’re more experienced than they are, you’ll never find out that they’re relying on fast uptake, rather than knowledge.

  • Thanatos Savehn

    What I’m wondering is whether post-persons ( will be better or a whole lot worse about protecting their status by protecting their hypotheses from critical and sceptical inquiry.

  • Jim Manzi


    Thanks for your detailed comments.

    WRT your specific questions:

    “1) why firms are willing to pay such huge sums for analyses of price elasticity, costing, and competitor responses that they could seemingly do themselves cheaper”?

    They can’t do it themselves. They are not paying for commoditizied analyses of these types, but these are typically integrated into work that is non-commoditized and creates insights that generate profit that they would not get on their own. At least this has been my experience in thestrategy consulting projects that I led.

    “2) why consulting firms seem so much more eager than other firms to pay steep premiums to hire fresh top school grads, when lesser school grads seem nearly as capable and much cheaper, and more experienced folks would seem more useful.”?

    They are not nearly as capable (in an economic sense). A portfolio of people who (i) score an average of 1565 on the M+V SATs,(ii) have an average GPA of 3.75 from top universities in math, physics, computer science or similar majors, and (iii) have survive a series of typically 12 – 15 analytical interviews that ~25% of the people with (i) and (ii) can survive (this is the profile of the typical hired class of organizations that I have led) can achieve things that others can’t. In my view, the economic value of person X in this kind of work does show marginal declines to increasing levels of this kind of talent, but the reverse. They get paid for that.

    In my experience, these same people typically have multiple offers from strategy consulting an/or high-end tech and/or the flavor du jour on Wall Street. Generally, the other offers are at least competitive with the consulting comp, indicating that there is a broad market for their labor that prices them this high.

    If this sounds like too much chest-pounding, please read my detailed post on this, which is careful to say these people are not “the best and brightest” in some absolute sense, they just have a specific set of characteristics that are worth a lot of money right now.

    • Mark M

      If they aren’t the best and brightest, why are they worth more than the best and brightest?

      My guess would be signalling. Even non-consulting organizations send signals, internally and externally, by hiring high status grads.

      • Jim Manzi

        What I meant was that they are “the best and brightest” in some general sense. They have a set of capabilities that in our current time and place have high economic value.

      • Jim Manzi

        Typing quickly…prior comment should have read “aren’t”

        Obviously, I’m not either…LOL

      • Mark M

        I understand that they have high economic value. The question is why?

        Let me try to ask the question more completely. Why would a fresh Ivy-league grad command a higher salary than a fresh State U grad who is objectively shown to be “better and brighter?”

        It’s not raw objective performance, since the question assumes the State U grad, in this specific case, is better and brighter. My answer is signalling. What’s yours?

      • Jim Manzi

        What I mean is that person X who can add tremendous value in the America of 2012 wouldn’t necessarily have been able to add nearly as much value relative to others in the America of 1950, or the England of 1850 or the America of 2050.

      • Jeff S

        Where are these mythical State U graduates who are objectively “better and brighter?” How many State U grads have M+V of 1565 (never mind the other criteria he mentioned)? They only exist at a few elite State U’s and those schools do get some hiring attention by the consulting firms.

        Even if there are a few of these hidden gems out there somewhere, they are probably hard to find. The cost-to-hire (when you have to go through 100 candidates to find one hire) may be higher even if the salaries end up being lower.

      • Mark M

        Jim –

        You’re answer doesn’t make sense to me. It seems to avoid the question. Please don’t mistake me – I’m not trying to imply good, bad, right, or wrong with my line of questioning. “Best and brightest,” which I take as a measure of potential job performance in the given field of study, is not the only criteria we use to measure the value of graduates. To throw one example out – a positive attitude can be infectious and may improve performance of others, making that individual more valuable. Back to the question, rephrased yet again but still the question I intended: What value does a high status school add beyond that available in a low status school? (I’m not arguing there is no value – my belief is the value is the status itself and the signal it sends. Signals are highly valuable).

        Jeff S –

        My line of questioning is based on Jim’s final paragraph above, where he states that these people are not necessarily the best and brightest, but have a certain set of characteristics that are valuable right now. I am not arguing these points – I’m asking what these characteristics are, and why high status universities infuse their graduates with those characteristics while low status universities do not. Recruitment costs may be a factor, but does not seem sufficient to explain the difference in value placed on these grads.

        As far as where to find these hidden gems? If I could tell you, they wouldn’t be hidden. My guess is just about anywhere. My test scores were easily high enough to get me into the most selective schools in the nation. I studied at the University of Iowa because that’s where I lived and worked at the time. (I’m not the exception that proves the rule – I’m the exception that proves there are exceptions). High achievers are far more prevalent at more selective schools, but it’s likely these hidden gems are scattered all around you.

      • Jeff S

        Mark, I would argue that the value the top schools are adding is partially status, but to a large extent sorting (in their admissions process) and testing. Test scores matter, but people who get through those programs (and a few people do drop out and many end up with less demanding majors) are succeeding *in competition with other very, very strong students*. This was very likely not the case so much for you at Iowa.

        Also, much of the recruiting is done on campus. Even though there may be a few hidden gems at each school, and indeed possibly a lot of them scattered around, does it make sense to seek them out at U/Iowa? Probably not.

        And this is not to take anything away from you personally, I don’t know you, of course. In the abstract it probably says something about someone who goes to an inferior school because it is good enough and happens to be located where he or she already “lived and worked”. That suggests less ambition and willingness to sacrifice for excellence. This trait probably worth something.

      • Mark M

        Jeff S –

        Rest assured, it was expected return on investment (yes, I did that analysis) rather than lack of ambition that kept me in Iowa. I already had a well-paying job when I decided to go to grad school.

    • The limited number of top school grads must of course be different in some way, and they must go somewhere, and yes they don’t all go to the same place. But since they are a small fraction of such jobs, there must be something unusual about those jobs. My suggestion is that such jobs have an especially high gain from status. I was asking for your explanation of what is unusual about management consulting jobs.

      • Jim Manzi


        My point in one of the paragraphs of my comment was that management consulting really isn’t that unusual in terms of total first year comp for new graduates.

        The average number of competitive job offers for one of our offerees was usually about 3 in most years. Across a portfolio of offerees, this would typically include offers from competing strategy firms, Wall Street jobs and high-end software engineering. In a normal year, the median consulting offeree had at least one higher-paid offer from a non-consulting company.

        Wall Street job paid more, initially and over time, but had more risk, worse lifestyle and were less intellectually interesting. Software paid more up-front, less over time, and the relative lifestyle and intellectual appeal depended on the interests of the candidate

      • I didn’t say these jobs paid more for the same employee features, I suggested that these employers have a much stronger taste for new grads of top schools. That unusual taste is what is in need of explanation.

      • Jim Manzi

        The relevant parts of finance and tech compete aggressively for the same top grads.

      • But what predicts which parts of any industry will be especially interested in new grads of top schools? It is a small fraction of jobs that are filled by this small fraction of workers, so it must be something very unusual about those jobs.

      • Mark M

        Jim –

        I think you’re dancing around the idea that management consulting firms value the status of their recruits and are willing to pay a premium for acquiring that status. I don’t see that you’ve denied this, but you haven’t admitted it either.

      • Jim Manzi


        My view is that these parts of these industries (e.g., the quant hedge funds in finance, the strategy consulting part of consulting, the very high-growth and high revenue multiple parts of software, etc.) are parts of industries that can use extremely high talent along some ix of dimensions of analytical intelligence, propensity for very long hours of work on a sustained basis, etc. as inputs into producing goods and services they can sell to customers at very high economic profits.


        I did a very long post on this (that may be the one Robin originally linked to). One of the points I made is that the leading strategy firms recruit at about 40 – 50 universities. So, as a starting point, if you go to say UCLA or Notre Dame or the University of Michigan, get very high grades in a very difficult major and score extremely well on standardized tests, you can go through the recruiting process right on campus. If you go to basically any school and get a 4.0 in math, physics or a similar major and score above 1550 on the SATs, you can likely get an interview by writing to the firm. If you did none of these things, but then get into any one of the top 10 or 15 MBA programs, you can then interview right on campus. It’s not prestige per se that matters; it’s predictors of job performance. And yes, the firms build prediction models that relate knowable data prior to hiring to subsequent performance.

  • Adrian Ratnapala

    *but the above response seems to miss the point – authority and accountability won’t be decentralized if that lowers the status of central folks. Just because they should decentralize doesn’t mean they will.*

    I don’t get this. Let’s say that JM is trying change, rather than change the future. Then the best guess is that, at the margin, arguing for decentralisation will make more of it happen than if he didn’t argue for it. And if he is right that the decision to decentralise will improve the quality of other decisions, then JM is wise to spend his limited pull by “going meta”. If he is right, whatever small amount of decentralisation he creates will have a large multiplier.

    • Adrian Ratnapala

      Err. That should have been “change, rather than predict.”

      Not “…, rather than change”.

      I’m sure there are other errors of language in my comment, but that one seemed like a biggie.

  • Mark M

    Manzi’s underlying assumption is incorrect. He should have quoted Ghostbuster’s:

    I liked the University. They gave us money, they gave us the facilities and we didn’t have to produce anything! I’ve worked in the private sector. They expect results.

    Unfortunately, most of us in the real world aren’t paid per performance or the value of our work output. Yes, a certain minimum performance is required, but we’re usually paid based on market value for our profession and our relative status within the demographic. Our status is usually determined more by signalling than by an objective measure of performance. So, yes, in the real world once a certain level of performance is achieved, signalling is given a much higher priority than higher performance.

    Decentralization could actually allow a greater divide between objective performance and perceived performance. Decentralizing authority and accountability doesn’t help until performance is measured objectively and rewarded fairly. Historically, we’ve been bad at this.

  • Drewfus

    Perhaps consultancy is about overcoming project design timing issues. Specifically, once a project is announced, the scope for creative design is minimized, because the project definition constrains this.

    Bringing in the highly paid consultants gives the excuse required to go “back to the drawing board.” The high pay motivates the company to get value for money from the process. The prestigious credentials of the consultants are partly the excuse required for the high pay, but there is also, at the very least, the benefit of a ‘second opinion’. So the high pay drives the company, as much as it drives supply.

    Another demand-side fix.

  • Annonymous

    Analyses and speculation aside, let me just be perfectly clear: the majority of projects firms like McKinsey and BCG engage their clients on are value-destroying projects. They are value destroying for the following reasons:

    A) What little competitive intelligence consulting firms can provide is mitigated in the long-run by the fact that these firms are also gathering intelligence on YOU once you’ve hired them.

    B) These firms have leveraged their business model to the maximum possible limit in order to drive revenues. What I mean is that a project leader – the only consultant with real experience/knowledge – will spend a maximum of 10-15% of his or her work time to any one project. The vast majority of the work is, in fact, done by freshly minted Ivy League undergrads and MBAs.

    C) The intellectual horsepower undoubtedly possessed by the folks hired by these consulting companies is mitigated entirely by the very human fact that clients basically just want to confirm their preconceived notions and biases about appropriate ‘next steps.’

    Point C) implies that consulting firms are not the ones destroying values, but it is actually the decision of value-destroying managers that creates the problem – but nonetheless…

    The greatest irony about the consulting phenomenon is that I hear a lot of consultants (I work for McKinsey) talk about how ‘stupid’ or ‘unsophisticated’ or ‘helpless’ their various clients are. The question I beg to them is: what would you expect? All of the smart people in our economy go into management consulting, and none of them go into industry. So you’re in effect blaming a client for a problem you yourself have created for client companies (withholding your own skills as a dedicated employee). Just my $0.02.

    • Anon

      10-15%? Not sure what type of stuff you do, but in my experience, we usually have experts for 25-50% – often very much on the ground, too.

      As for the helpless clients – maybe if they were to develop career tracks with less politics then consultants would chose to go industry instead. Having seen politics in your average big company up close times and times again I have fairly limited desire to go there.

  • Collusion

    If consultants from one company both get farmed out to two competing firms, they can collude to raise prices without a competitive response, making money for both firms and stiffing customers.