Incomplete Analysis

In reading the comments on my variance-induced test bias post, I was reminded of a big bias loophole in social science: judging when an analysis is complete "enough."  We usually have some status quo policies, and some analyses relevant to those policies.  Each analysis tends to favor some possible policies relative to others, but alas most every analysis is incomplete, leaving out relevant considerations. 

Now we do need to assess which analyzes are most relevant to any given policy question, but at least here experts can, when analyses are similar enough, usually bring to bear some relatively "objective" criteria.  When we ask if the relevant analyses are good "enough" to justify action, however, we can usually appeal only to much weaker standards of evaluation. 

Those who like the implications of current analyses may insist that honesty demands we act on their recommendations, while those who dislike those recommendations may say analysis incompleteness means we just do not have enough evidence to justify changing our behavior.  For example, those who like the idea of having college admission boards adjust SAT scores as I suggested may say my analysis is plenty detailed enough, while those who dislike this policy may say we need far more detailed analyses before we could even consider such a policy. 

In such a situation it can be very hard for observers, or even participants, to know which side to believe.  Whatever are our most detailed current analyses, those who think their recommendations misleading can usually point to crucial considerations which, if included in a more complete analysis, might well overturn those recommendations.  But such critics often feel under no obligation to actually produce such more complete analyses.

There are many possible standards we could apply here:

  1. we follow the most detailed analyses available, even if very incomplete
  2. we follow the analyses preferred by the most prestigious academics available
  3. we follow the advice of decision markets, regardless of trader analysis detail
  4. we together elect a representative who then judges for him/herself. 
  5. each person judges for him/herself; no more common standard is desirable

Which standard is best in what circumstances? 

GD Star Rating
Tagged as: ,
Trackback URL:
  • It seems to me that everything rational is recursive ultimately to the decision of experts. If it’s not experts on the specific topic decided upon, than it’s experts in decision markets, or an expert in the democratic process, or an expert in the degree to which each person is being allowed to decide for themself. So I think that de-exceptionalizes the closest thing you have to an expression of that, in rather warped form, as your #2.

    I lean towards the consensus of subject matter experts, who are free to use 1-5, mediated by decision theory experts. I think such decisions should be subject to periodic review and new data comes in, and that where possible their should be regional experimentation to help produce such data (even to the extent of double blind trials and all that).

  • I hate to do anything as lame as correct spelling, but I think you mean “analyses” when you write “analyzes”.

  • A prediction market that is known to drive a decision will fail. Proponents of one or both courses of action will flood it with money based on their beliefs, and consider it a “donation” if it fails.

    Imagine a market set up to predict the potential effects of climate change, that is directly tied into the level of a carbon cap. It would quickly become a “vote with your wallet” poll. Many would view the expenditure as a donation rather than an investment in a prediction.

    Sadly, this isn’t too far from our actual political system, and at least the people who are right would see some benefit, rather than only the advertising companies. Still, ideological donations could weaken any prediction markets given decision force.

  • Hopefully, academic experts are chosen primarily for producing impressive analyses, and impressiveness is only somewhat related to accuracy and completeness.

    Allan, Typepad wanted on that spelling; yet another reason to dislike Typepad.

    Gavin, we have lots of data and theory on such manipulation, suggesting it isn’t a problem.

  • number 5 seems to allow for most options, since individuals deciding for themselves could choose to hire represenatives, enact decision markets, follow the most detailed analysis, et c. in deciding what policy to enact.

    i’m not sure what would be best regarding choices that require consensus. “vote on values, bet on beliefs” seems attractive, so represenatives and prediction markets would be the attractive decision tools–prediction markets would make use of various analyses out there in the world, presumably.

    but to be honest, if the values voted on run counter to my own values, i’m disinclined to go with the consensus, and i suspect many, perhaps most people think along these lines. voting seems an attractive means compared to others of resolving conflict, but not always. ‘vote when it doesn’t seriously move us away from advancing my libertarian values, or when it will advance my libertarian values as an aim of government, bet on beliefs regarding how to best advance those libertarian values.’ that might be my verbose motto.

  • Could you link to your proposal on SAT scores?

  • Nevermind, I thought that was something you had talked about earlier than the variance-induced bias post.

  • Robin, excellent paper. Your data turns my point around. This is a great way to make money off of people with incorrect beliefs–what better way to drain the coffers of the biased and foolish?

    There is still the problem that if a prediction market is linked to action, the prediction can’t be something that would be altered by that action.

    Of course if one of the outcomes of the prediction could affect the market itself, there’s another problem. There would be no point in better on global disaster, with no way to realize the benefits of winning.

    I imagine that these issues have come up in other papers so I’ll stop there.

  • Robin, it doesn’t seem to me like you’re setting “prediction markets” against the strongest alternatives if you put up a version of “consensus of experts” that you admit is a weakened representation: “we follow the analyses preferred by the most prestigious academics available”, because you claim “academic experts are chosen primarily for producing impressive analyses, and impressiveness is only somewhat related to accuracy and completeness”.

    For example, how about the consensus of experts chosen primarily for producing analysis that is most accurate?