Selection Bias in Economic Theory

A bit ago I surprised David Balan by suggesting that selection bias is so strong that when estimating the health effect of something like alcohol, we should prefer the sloppier and noisier control variable estimates from papers that focus on other topics, relative to the estimates from papers where alcohol is the main focus. 

Selection bias is also very strong in economic theory.   Here also, "authors, funders, and referees have answers they expect and want to see," and authors can search among possible assumptions to find models that give expected answers.  Unfortunately, there is no useful theory analog of bias-avoiding control variable estimates.  So how can we avoid selection bias in policy advice from economic theorists?

The clearest way I can see is to limit attention to a small range of standard theoretical models, a range that is still capable of giving clear policy advice covering the full ideological advice range.  And the obvious choice here is:  economic efficiency evaluations of supply and demand with externalities and transaction costs.   This is the main framework taught in introductory economics courses, and is fully capable of recommending high levels of regulation and intervention, depending on what empirical findings one applies.

Of course professional economic theorists such as myself know a lot more than this basic theory, and it seems a terrible shame to ignore all this further insight.  But I’ll have to admit we are so capable of choosing further assumptions to get the answers we want that outsiders can’t gain much policy advantage from our further insight.  I have a distant hope that betting markets can someday help us overcome this serious limitation. 

GD Star Rating
a WordPress rating system
Tagged as:
Trackback URL:
  • EthanJ

    In the short term, how about more peer review earlier in the process? Before research progresses, experimental design (or methodology) should always be discussed with others to ensure the most robust results. If researchers were required to explicitly seek the advise of those who disagree, or at least of a sufficiently large panel of their peers, it would force researchers to defend their choices of control variables and methodology.

    The scientific method as usually taught involves hypothesis and experimental design BEFORE data collection and conclusions. If bias creeps in because experimental design is not being sufficiently analyzed and researchers are deceiving themselves as to the appropriateness of their choices, perhaps we need to make the steps in the method more discrete.

  • Nick Bostrom

    Ethan, your comment makes me think it would be cool to have more co-authored papers, written by two people or groups known to take opposing stands on some issue, and presenting the outcome of some experiment that resolved their disagreement. They would collaborate in designing a suitable test, agree in advance what would count as a “win” for one side or the other; then conduct the study and report the results.

    In non-experimental subjects one could do something similar: for example, two philosophers starting with opposite views would work on a paper together and report whatever arguments they found that led at least one of them to change her mind about something of relevance. A kind of “significance test” for a new philosophical argument could be that it would have succeeded in changing at least one other philosopher’s opinion…

  • Robin Hanson

    Nick, an intriguing suggestion, but it will often be hard to calibrate the previous position of a person on a subject. Does the person who changed their mind about something no longer count as either side for future such papers?

  • Nick Bostrom

    Robin, I was thinking of this as an ad hoc approach, which will work in some cases, and which it would be nice to see done a bit more often than is currently the case. I’m not sure whether or how it could be developed into a proposal for more systematic change. The stance of at least some experts are often well known within small expert communities. I guess a person who changed her mind could again count as a side, but the result is likely to be impressive in proportion as the persuaded party is known to have had a long-standing and well-informed committment to the abandoned position.

  • David J. Balan

    As you pointed out, there is a tradeoff built into this idea: richness of model vs. scope for hiding the ball. But there is another problem. In those settings where the analysis is going to have some kind of policy significance, having the world know that the analyst is going to use a particular simple model no matter what will invite attempts to game the system.

  • Robin Hanson

    David, what games did you have in mind?

  • Jason

    I think I have some idea what David is referring to. Competing interest groups choose arguments that support their preferred policy; restricting the class of valid and accepted arguments (which is what this suggestion does) increases the importance of the acceptable arguments, and hence the willingness of those interest groups to influence those specific arguments toward their desired policy. (Making simpler arguments also makes it easier to see how to influence those arguments to get a desired result, though I doubt this is hard as is.)

    A clarifying example: One way to “game the system” is to input misleading or inaccurate statistics into the model. This may not be a problem if the original source of the statistics is unbiased, but this isn’t always true; e.g., crime statistics come from police departments. (And in any case, the model makers always have to select a statistic, another source of bias.) If the model is largely known in advance it is easier to predict the effect of changes in the statistics, hence easier to manipulate them (consciously or unconsciously) to get the “correct” policy result.

  • Robin Hanson

    Jason, if you can select different statistical methods or sources, you can just keep trying them to get the result you want, no matter how complicated the theoretical model is. You just need some automated way to plug your stat results into the model.