Comments on: Control Variables Avoid Bias http://www.overcomingbias.com/2007/02/control_variabl.html Overcoming Bias is economist Robin Hanson’s blog, on honesty, signaling, disagreement, forecasting, and the far future. Sat, 11 Feb 2012 23:23:58 +0000 hourly 1 http://wordpress.org/?v=3.1.3 By: Robin Hanson http://www.overcomingbias.com/2007/02/control_variabl.html#comment-422212 Robin Hanson Fri, 16 Feb 2007 11:40:31 +0000 http://prod.ob.trike.com.au/2007/02/control-variables-avoid-bias.html#comment-422212 Douglas, yes of course it would be better to know in more detail who is biased in which direction. Conchis, the point is that it is much harder to game all the control variable estimates. Douglas, yes of course it would be better to know in more detail who is biased in which direction.

Conchis, the point is that it is much harder to game all the control variable estimates.

]]> By: conchis http://www.overcomingbias.com/2007/02/control_variabl.html#comment-422211 conchis Fri, 16 Feb 2007 11:04:00 +0000 http://prod.ob.trike.com.au/2007/02/control-variables-avoid-bias.html#comment-422211 If we all started taking this advice, wouldn't we then expect researchers to just start gaming the system? If we all started taking this advice, wouldn’t we then expect researchers to just start gaming the system?

]]> By: Douglas Knight http://www.overcomingbias.com/2007/02/control_variabl.html#comment-422210 Douglas Knight Wed, 07 Feb 2007 17:38:15 +0000 http://prod.ob.trike.com.au/2007/02/control-variables-avoid-bias.html#comment-422210 How would one test the claim that selection bias is worse than other biases (failure to consider endogeneity, I guess)? RH's observation that you get different answers is a necessary condition, but not sufficient. If you can exhibit explicit biases, like correlation with funding, that's pretty good, but it might be that one of the sides is doing the appropriate correction and the other isn't. Moreover, if you can exhibit explicit biases, you probably have better options available. How would one test the claim that selection bias is worse than other biases (failure to consider endogeneity, I guess)? RH’s observation that you get different answers is a necessary condition, but not sufficient. If you can exhibit explicit biases, like correlation with funding, that’s pretty good, but it might be that one of the sides is doing the appropriate correction and the other isn’t. Moreover, if you can exhibit explicit biases, you probably have better options available.

]]> By: Robin Hanson http://www.overcomingbias.com/2007/02/control_variabl.html#comment-422209 Robin Hanson Wed, 07 Feb 2007 15:26:37 +0000 http://prod.ob.trike.com.au/2007/02/control-variables-avoid-bias.html#comment-422209 David, yes, my best guess is that empirical selection bias is so bad that we are better off on relying on a mix of control variable estimates and randomized experiments, with a heavier emphasis on experiments that one would otherwise choose. David, yes, my best guess is that empirical selection bias is so bad that we are better off on relying on a mix of control variable estimates and randomized experiments, with a heavier emphasis on experiments that one would otherwise choose.

]]> By: David J. Balan http://www.overcomingbias.com/2007/02/control_variabl.html#comment-422208 David J. Balan Wed, 07 Feb 2007 15:03:52 +0000 http://prod.ob.trike.com.au/2007/02/control-variables-avoid-bias.html#comment-422208 Robin, I realize that could be true. Are you saying that things as they are are bad enough that it actually is true? If so, that's a pretty radical statement. Robin, I realize that could be true. Are you saying that things as they are are bad enough that it actually is true? If so, that’s a pretty radical statement.

]]> By: Robin Hanson http://www.overcomingbias.com/2007/02/control_variabl.html#comment-422207 Robin Hanson Wed, 07 Feb 2007 10:47:05 +0000 http://prod.ob.trike.com.au/2007/02/control-variables-avoid-bias.html#comment-422207 David, yes, when selection bias is bad enough, we are better off with only OLS biases. David, yes, when selection bias is bad enough, we are better off with only OLS biases.

]]> By: David J. Balan http://www.overcomingbias.com/2007/02/control_variabl.html#comment-422206 David J. Balan Wed, 07 Feb 2007 03:53:10 +0000 http://prod.ob.trike.com.au/2007/02/control-variables-avoid-bias.html#comment-422206 Empirical researchers generally only seriously investigate one variable at a time. A million papers have been written trying to estimate the returns to education, and a million more to estimate the returns to tenure. But the ones that focus on education just throw tenure in on the right-hand side of the regression and vice-versa. I've certainly been guilty of this, and I'm not enough of an econometrician to know what to do about it, or if anything can be done. But it does suggest a problem with Robin's approach. The right-hand side variables that the researcher is not focused on were probably just thrown in there, without any attempt to deal with endogeneity or anything else. So while the estimates of those coefficients are free of one kind of bias (the bias in favor of whatever the researcher wants the answer to be), they may have other biases that are worse. It seems like Robin's suggestion is tantamount to suggesting that we would be better off if all researchers were constrained to just always run OLS. Isn't it? Empirical researchers generally only seriously investigate one variable at a time. A million papers have been written trying to estimate the returns to education, and a million more to estimate the returns to tenure. But the ones that focus on education just throw tenure in on the right-hand side of the regression and vice-versa. I’ve certainly been guilty of this, and I’m not enough of an econometrician to know what to do about it, or if anything can be done. But it does suggest a problem with Robin’s approach. The right-hand side variables that the researcher is not focused on were probably just thrown in there, without any attempt to deal with endogeneity or anything else. So while the estimates of those coefficients are free of one kind of bias (the bias in favor of whatever the researcher wants the answer to be), they may have other biases that are worse. It seems like Robin’s suggestion is tantamount to suggesting that we would be better off if all researchers were constrained to just always run OLS. Isn’t it?

]]> By: Robin Hanson http://www.overcomingbias.com/2007/02/control_variabl.html#comment-422205 Robin Hanson Wed, 07 Feb 2007 01:06:25 +0000 http://prod.ob.trike.com.au/2007/02/control-variables-avoid-bias.html#comment-422205 Perry, I would prefer journals that followed your rule, as well as the rule that clinical trials must be declared beforehand, to let us correct for publication selection bias. But in the competitive market for journals, it is not clear our preferences will win. Perry, I would prefer journals that followed your rule, as well as the rule that clinical trials must be declared beforehand, to let us correct for publication selection bias. But in the competitive market for journals, it is not clear our preferences will win.

]]> By: Perry E. Metzger http://www.overcomingbias.com/2007/02/control_variabl.html#comment-422204 Perry E. Metzger Wed, 07 Feb 2007 00:53:48 +0000 http://prod.ob.trike.com.au/2007/02/control-variables-avoid-bias.html#comment-422204 I think that, now that we have the internet, people should be required to provide their raw data and calculations in supplementary online documentation accompanying every journal article. We are no longer in an age where there is an excuse that there is no place to store the information, and it would be much easier for people to check other people's work if the entire corpus was made available. I think that, now that we have the internet, people should be required to provide their raw data and calculations in supplementary online documentation accompanying every journal article. We are no longer in an age where there is an excuse that there is no place to store the information, and it would be much easier for people to check other people’s work if the entire corpus was made available.

]]> By: Robin Hanson http://www.overcomingbias.com/2007/02/control_variabl.html#comment-422203 Robin Hanson Tue, 06 Feb 2007 18:51:07 +0000 http://prod.ob.trike.com.au/2007/02/control-variables-avoid-bias.html#comment-422203 Nick, usually: medical spending and water supply have no effect, alcohol is good, low weight is bad, and rural living is good. Nick, usually: medical spending and water supply have no effect, alcohol is good, low weight is bad, and rural living is good.

]]>