A Wonk’s First Question

Imagine you are considering a career as a paid policy wonk. You wonder which policy area to work in, and which institutions to affiliate with. If you want to influence actual policy, rather than just enjoying money and status, your should ask yourself: do those who sponsor or cite efforts in this area do so more to get support for pre-determined conclusions, or more to get info to help them make choices?

Sometimes sponsors and other consumers of a type of policy analysis know what policies they prefer, but seek prestigious cover to help them do it. So they pay and search for policy analyses in the hope of finding the support they need. With enough policy analyses to choose from, they can find ones to support their predetermined conclusions. But they need these prestigious cover analyses to appear, at least to distant observers, to be honest open attempts at discovery. It can’t be obvious that they were designed to support pre-determined conclusions.

At other times, however, sponsors and consumers are actually uncertain, and seek analyses with unpredictable-to-them conclusions to influence their choices. And these are the only cases where your being a policy analyst has a chance of changing real policy outcomes. Such audiences may see your analysis, or be influenced by someone else who has seen them. So for each analysis that you might produce, you should wonder: what are my chances of influencing such an open-minded chooser?

Here are a few clues to consider:

  1. How predictable are the policy conclusions of the most popular policy analysts in this area? High predictability suggests that sponsors reward such consistency, as it aids their efforts to collect support for predetermined conclusions.
  2. How interested are sponsors and other policy consumers in policy analyses done by very prestigious people and institutions, relative to others? The more open they are to products of low prestige analysts, the better the chance they seek information, instead of just prestigious backing for pre-existing conclusions.
  3. How open is this area to funding and otherwise supporting large relevant experiments (or prediction markets)? Or to applying a strong standard theory with standard assumptions, which together often imply specific conclusions? The more that people are willing to endorse the policy implications of such things before their results become known, the more open that area is to unpredictable new information.

It should be possible to collect evidence on how these factors vary across policy areas. Perhaps a simple survey would be sufficient. Might that publicly reveal to all the relative sincerity of different kinds of sponsors and consumers of policy analysis?

GD Star Rating
loading...
Tagged as: ,
Trackback URL:
  • http://invariant.org/ Peter Gerdes

    I’m not sure #2 really pushes in the direction you suggest. If I really have no idea about a subject or how to evaluate it then paying attention to anyone but the most prestigious opinions in the area is suggestive of cherry picking.

    Being open to analysis by low prestige institutions and individuals presupposes the ability of the consumer to evaluate the quality of the analysis themselves which often isn’t the case.

    For instance, I think most politicians when deciding how to allocate money between various space missions or particle physics experiments would be best served by only listening to the opinions of the most prestigious institutions/members in the astronomy and physics community simply because they have no way of evaluating for themselves whether any particular proposal is worthwhile or not.

    • http://overcomingbias.com RobinHanson

      I didn’t say that this is the only reason one might focus on prestige. But all else equal it adds.

      • http://invariant.org/ Peter Gerdes

        Well I’m not sure how the frequencies break down (more often looked at for good reasons or bad reasons) but I would certainly agree it is a clue to look for and once noticed I think we can usually distinguish the cases prestige is playing a strong role in choice of advisors at for appropriate and inappropriate reasons. So I guess I’m just nitpicking. I find the overall thrust of your post convincing.

        I do worry that, as a practical matter (and nothing to do with your suggestions) all the subtlety here (sometimes prestigious advisors are chosen for perfectly normal reasons) provides people a great deal of opportunity to let the very kind of bias you warn about influence their judgement of whether bias exists in that area. Just remarking that it is, of course, a hard thing to do consistently not disagreeing.

  • http://invariant.org/ Peter Gerdes

    Also, I’m not sure that 1 is quite right either. Seems to me what matters isn’t so much predictability but whether or not the policy analysis agrees with peoples (and policy consumers) prior expectations.

    For instance, it is very predictable that the most respected economic policy analysts will almost always recommend more free trade and less tariffs. However, the fact that this is *so* predictable and yet so frequently in tension with peoples (including policy consumers) prior expectations is evidence against the idea that this conclusion is driven by the fact that sponsors reward that kind of consistency.

    • http://overcomingbias.com RobinHanson

      If you have data on the answers that sponsors and consumers of policy analysis want to get, yes of course you can use that clue as well.

  • https://llordoftherealm.wordpress.com/ Lord

    Since when do sponsors need the appearance of objectivity? It can help with effectiveness of persuasion and result, but if you have already decided, it can get in the way of selling with inconvenient facts. Marketing is marketing. You would be hack than wonk but hack generally pays better with less work and is at least as much in demand.

  • Riothamus

    Alas and alack, each clue is wonky by itself and probably not the same area of expertise one is considering as a base for their career.