A Wonk’s First Question
Imagine you are considering a career as a paid policy wonk. You wonder which policy area to work in, and which institutions to affiliate with. If you want to influence actual policy, rather than just enjoying money and status, your should ask yourself: do those who sponsor or cite efforts in this area do so more to get support for pre-determined conclusions, or more to get info to help them make choices?
Sometimes sponsors and other consumers of a type of policy analysis know what policies they prefer, but seek prestigious cover to help them do it. So they pay and search for policy analyses in the hope of finding the support they need. With enough policy analyses to choose from, they can find ones to support their predetermined conclusions. But they need these prestigious cover analyses to appear, at least to distant observers, to be honest open attempts at discovery. It can’t be obvious that they were designed to support pre-determined conclusions.
At other times, however, sponsors and consumers are actually uncertain, and seek analyses with unpredictable-to-them conclusions to influence their choices. And these are the only cases where your being a policy analyst has a chance of changing real policy outcomes. Such audiences may see your analysis, or be influenced by someone else who has seen them. So for each analysis that you might produce, you should wonder: what are my chances of influencing such an open-minded chooser?
Here are a few clues to consider:
How predictable are the policy conclusions of the most popular policy analysts in this area? High predictability suggests that sponsors reward such consistency, as it aids their efforts to collect support for predetermined conclusions.
How interested are sponsors and other policy consumers in policy analyses done by very prestigious people and institutions, relative to others? The more open they are to products of low prestige analysts, the better the chance they seek information, instead of just prestigious backing for pre-existing conclusions.
How open is this area to funding and otherwise supporting large relevant experiments (or prediction markets)? Or to applying a strong standard theory with standard assumptions, which together often imply specific conclusions? The more that people are willing to endorse the policy implications of such things before their results become known, the more open that area is to unpredictable new information.
It should be possible to collect evidence on how these factors vary across policy areas. Perhaps a simple survey would be sufficient. Might that publicly reveal to all the relative sincerity of different kinds of sponsors and consumers of policy analysis?
Alas and alack, each clue is wonky by itself and probably not the same area of expertise one is considering as a base for their career.
Well I'm not sure how the frequencies break down (more often looked at for good reasons or bad reasons) but I would certainly agree it is a clue to look for and once noticed I think we can usually distinguish the cases prestige is playing a strong role in choice of advisors at for appropriate and inappropriate reasons. So I guess I'm just nitpicking. I find the overall thrust of your post convincing.
I do worry that, as a practical matter (and nothing to do with your suggestions) all the subtlety here (sometimes prestigious advisors are chosen for perfectly normal reasons) provides people a great deal of opportunity to let the very kind of bias you warn about influence their judgement of whether bias exists in that area. Just remarking that it is, of course, a hard thing to do consistently not disagreeing.