A bit ago I surprised David Balan by suggesting that selection bias is so strong that when estimating the health effect of something like alcohol, we should prefer the sloppier and noisier control variable estimates from papers that focus on other topics, relative to the estimates from papers where alcohol is the main focus.
Jason, if you can select different statistical methods or sources, you can just keep trying them to get the result you want, no matter how complicated the theoretical model is. You just need some automated way to plug your stat results into the model.
I think I have some idea what David is referring to. Competing interest groups choose arguments that support their preferred policy; restricting the class of valid and accepted arguments (which is what this suggestion does) increases the importance of the acceptable arguments, and hence the willingness of those interest groups to influence those specific arguments toward their desired policy. (Making simpler arguments also makes it easier to see how to influence those arguments to get a desired result, though I doubt this is hard as is.)
A clarifying example: One way to "game the system" is to input misleading or inaccurate statistics into the model. This may not be a problem if the original source of the statistics is unbiased, but this isn't always true; e.g., crime statistics come from police departments. (And in any case, the model makers always have to select a statistic, another source of bias.) If the model is largely known in advance it is easier to predict the effect of changes in the statistics, hence easier to manipulate them (consciously or unconsciously) to get the "correct" policy result.
As you pointed out, there is a tradeoff built into this idea: richness of model vs. scope for hiding the ball. But there is another problem. In those settings where the analysis is going to have some kind of policy significance, having the world know that the analyst is going to use a particular simple model no matter what will invite attempts to game the system.
Robin, I was thinking of this as an ad hoc approach, which will work in some cases, and which it would be nice to see done a bit more often than is currently the case. I'm not sure whether or how it could be developed into a proposal for more systematic change. The stance of at least some experts are often well known within small expert communities. I guess a person who changed her mind could again count as a side, but the result is likely to be impressive in proportion as the persuaded party is known to have had a long-standing and well-informed committment to the abandoned position.
Nick, an intriguing suggestion, but it will often be hard to calibrate the previous position of a person on a subject. Does the person who changed their mind about something no longer count as either side for future such papers?
Ethan, your comment makes me think it would be cool to have more co-authored papers, written by two people or groups known to take opposing stands on some issue, and presenting the outcome of some experiment that resolved their disagreement. They would collaborate in designing a suitable test, agree in advance what would count as a "win" for one side or the other; then conduct the study and report the results.
In non-experimental subjects one could do something similar: for example, two philosophers starting with opposite views would work on a paper together and report whatever arguments they found that led at least one of them to change her mind about something of relevance. A kind of "significance test" for a new philosophical argument could be that it would have succeeded in changing at least one other philosopher's opinion...
In the short term, how about more peer review earlier in the process? Before research progresses, experimental design (or methodology) should always be discussed with others to ensure the most robust results. If researchers were required to explicitly seek the advise of those who disagree, or at least of a sufficiently large panel of their peers, it would force researchers to defend their choices of control variables and methodology.
The scientific method as usually taught involves hypothesis and experimental design BEFORE data collection and conclusions. If bias creeps in because experimental design is not being sufficiently analyzed and researchers are deceiving themselves as to the appropriateness of their choices, perhaps we need to make the steps in the method more discrete.
Jason, if you can select different statistical methods or sources, you can just keep trying them to get the result you want, no matter how complicated the theoretical model is. You just need some automated way to plug your stat results into the model.
I think I have some idea what David is referring to. Competing interest groups choose arguments that support their preferred policy; restricting the class of valid and accepted arguments (which is what this suggestion does) increases the importance of the acceptable arguments, and hence the willingness of those interest groups to influence those specific arguments toward their desired policy. (Making simpler arguments also makes it easier to see how to influence those arguments to get a desired result, though I doubt this is hard as is.)
A clarifying example: One way to "game the system" is to input misleading or inaccurate statistics into the model. This may not be a problem if the original source of the statistics is unbiased, but this isn't always true; e.g., crime statistics come from police departments. (And in any case, the model makers always have to select a statistic, another source of bias.) If the model is largely known in advance it is easier to predict the effect of changes in the statistics, hence easier to manipulate them (consciously or unconsciously) to get the "correct" policy result.
David, what games did you have in mind?
As you pointed out, there is a tradeoff built into this idea: richness of model vs. scope for hiding the ball. But there is another problem. In those settings where the analysis is going to have some kind of policy significance, having the world know that the analyst is going to use a particular simple model no matter what will invite attempts to game the system.
Robin, I was thinking of this as an ad hoc approach, which will work in some cases, and which it would be nice to see done a bit more often than is currently the case. I'm not sure whether or how it could be developed into a proposal for more systematic change. The stance of at least some experts are often well known within small expert communities. I guess a person who changed her mind could again count as a side, but the result is likely to be impressive in proportion as the persuaded party is known to have had a long-standing and well-informed committment to the abandoned position.
Nick, an intriguing suggestion, but it will often be hard to calibrate the previous position of a person on a subject. Does the person who changed their mind about something no longer count as either side for future such papers?
Ethan, your comment makes me think it would be cool to have more co-authored papers, written by two people or groups known to take opposing stands on some issue, and presenting the outcome of some experiment that resolved their disagreement. They would collaborate in designing a suitable test, agree in advance what would count as a "win" for one side or the other; then conduct the study and report the results.
In non-experimental subjects one could do something similar: for example, two philosophers starting with opposite views would work on a paper together and report whatever arguments they found that led at least one of them to change her mind about something of relevance. A kind of "significance test" for a new philosophical argument could be that it would have succeeded in changing at least one other philosopher's opinion...
In the short term, how about more peer review earlier in the process? Before research progresses, experimental design (or methodology) should always be discussed with others to ensure the most robust results. If researchers were required to explicitly seek the advise of those who disagree, or at least of a sufficiently large panel of their peers, it would force researchers to defend their choices of control variables and methodology.
The scientific method as usually taught involves hypothesis and experimental design BEFORE data collection and conclusions. If bias creeps in because experimental design is not being sufficiently analyzed and researchers are deceiving themselves as to the appropriateness of their choices, perhaps we need to make the steps in the method more discrete.