Selection Bias in Economic Theory
A bit ago I surprised David Balan by suggesting that selection bias is so strong that when estimating the health effect of something like alcohol, we should prefer the sloppier and noisier control variable estimates from papers that focus on other topics, relative to the estimates from papers where alcohol is the main focus.
Selection bias is also very strong in economic theory. Here also, "authors, funders, and referees have answers they expect and want to see," and authors can search among possible assumptions to find models that give expected answers. Unfortunately, there is no useful theory analog of bias-avoiding control variable estimates. So how can we avoid selection bias in policy advice from economic theorists?
The clearest way I can see is to limit attention to a small range of standard theoretical models, a range that is still capable of giving clear policy advice covering the full ideological advice range. And the obvious choice here is: economic efficiency evaluations of supply and demand with externalities and transaction costs. This is the main framework taught in introductory economics courses, and is fully capable of recommending high levels of regulation and intervention, depending on what empirical findings one applies.
Of course professional economic theorists such as myself know a lot more than this basic theory, and it seems a terrible shame to ignore all this further insight. But I’ll have to admit we are so capable of choosing further assumptions to get the answers we want that outsiders can’t gain much policy advantage from our further insight. I have a distant hope that betting markets can someday help us overcome this serious limitation.
Jason, if you can select different statistical methods or sources, you can just keep trying them to get the result you want, no matter how complicated the theoretical model is. You just need some automated way to plug your stat results into the model.
I think I have some idea what David is referring to. Competing interest groups choose arguments that support their preferred policy; restricting the class of valid and accepted arguments (which is what this suggestion does) increases the importance of the acceptable arguments, and hence the willingness of those interest groups to influence those specific arguments toward their desired policy. (Making simpler arguments also makes it easier to see how to influence those arguments to get a desired result, though I doubt this is hard as is.)
A clarifying example: One way to "game the system" is to input misleading or inaccurate statistics into the model. This may not be a problem if the original source of the statistics is unbiased, but this isn't always true; e.g., crime statistics come from police departments. (And in any case, the model makers always have to select a statistic, another source of bias.) If the model is largely known in advance it is easier to predict the effect of changes in the statistics, hence easier to manipulate them (consciously or unconsciously) to get the "correct" policy result.