Here is a debiasing technique one might try: writing a hypothetical apostasy.  Remind yourself before you start that unless you later choose to do so, you will never have to show this text to anyone.

Imagine, if you will, that the world's destruction is at stake and the only way to save it is for you to write a one-pager that convinces a jury that your old cherished view is mistaken or at least seriously incomplete.  The more inadequate the jury thinks your old cherished view is, the greater the chances that the world is saved.  The catch is that the jury consists of earlier stages of yourself (such as yourself such as you were one year ago).  Moreover, the jury believes that you have been bribed to write your apostasy; so any assurances of the form "trust me, I am older and know better" will be ineffective.  Your only hope of saving the world is by writing an apostasy that will make the jury recognize how flawed/partial/shallow/juvenile/crude/irresponsible/incomplete and generally inadequate your old cherished view is.

(If anybody tries this, feel free to comment below on whether you found the exersise fruitful or not – but no need to state which specific view you were considering or how it changed.)

It doesn't seem you can simply plug your uncertainty into expected utility decision theory and crank the wheel; because many moral theories state that you should not always maximize expected utility.

Even if we limit consideration to consequentialist theories, it still is hard to see how to combine them in the standard decision theoretic framework.  For example, suppose you give X% probability to total utilitarianism and (100-X)% to average utilitarianism.  Now an action might add 5 utils to total happiness and decrease average happiness by 2 utils.  (This could happen, e.g. if you create a new happy person that is less happy than the people who already existed.)  Now what do you do, for different values of X?

The problem gets even more complicated if we consider not only consequentialist theories but also deontological theories, contractarian theories, virtue ethics, etc.  We might even throw various meta-ethical theories into the stew: error theory, relativism, etc.

I'm working on a paper on this together with my colleague Toby Ord.  We have some arguments against a few possible "solutions" that we think don't work.  On the positive side we have some tricks that work for a few special cases.  But beyond that, the best we have managed so far is a kind of metaphor, which we don't think is literally and exactly correct, and it is a bit under-determined, but it seems to get things roughly right and it might point in the right direction:

The Parliamentary Model.  Suppose that you have a set of mutually exclusive moral theories, and that you assign each of these some probability.  Now imagine that each of these theories gets to send some number of delegates to The Parliament.  The number of delegates each theory gets to send is proportional to the probability of the theory.  Then the delegates bargain with one another for support on various issues; and the Parliament reaches a decision by the delegates voting.  What you should do is act according to the decisions of this imaginary Parliament.  (Actually, we use an extra trick here: we imagine that the delegates act as if the Parliament's decision were a stochastic variable such that the probability of the Parliament taking action A is proportional to the fraction of votes for A.  This has the effect of eliminating the artificial 50% threshold that otherwise gives a majority bloc absolute power.  Yet – unbeknownst to the delegates – the Parliament always takes whatever action got the most votes: this way we avoid paying the cost of the randomization!)

The idea here is that moral theories get more influence the more probable they are; yet even a relatively weak theory can still get its way on some issues that the theory think are extremely important by sacrificing its influence on other issues that other theories deem more important.  For example, suppose you assign 10% probability to total utilitarianism and 90% to moral egoism (just to illustrate the principle).  Then the Parliament would mostly take actions that maximize egoistic satisfaction; however it would make some concessions to utilitarianism on issues that utilitarianism thinks is especially important.  In this example, the person might donate some portion of their income to existential risks research and otherwise live completely selfishly.

I think there might be wisdom in this model.  It avoids the dangerous and unstable extremism that would result from letting one’s current favorite moral theory completely dictate action, while still allowing the aggressive pursuit of some non-commonsensical high-leverage strategies so long as they don’t infringe too much on what other major moral theories deem centrally important.

But maybe somebody here has better ideas or suggestions for improving this model?

To this end, let me tentatively propose a classification scheme, organized around the sources of bias:

Type-I biases arise from the fact that our beliefs sometimes serve functions – such as social signaling – that can conflict with their navigational (truth-tracking) function.  For example, our tendency to overestimate our own positive attributes may be an example of a Type I bias.

Type-II biases arise from the shortcomings and flaws of our minds.  We are subject to various kinds of processing constraints, and even aside from these hard limitations we weren’t very successfully optimized for efficiency in abstract rationality even in contexts where no adaptive function interferes with the navigational function of our beliefs.  Type II biases can result from fast-and-frugal heuristics that compromise accuracy for speed and ease of use, or from various idiosyncratic features of our brains and psyches.  We can distinguish subtype-II(a) biases deriving from shortcomings general to the human psyche (availability bias?), and subtype-II(b) biases deriving from shortcomings specific to some individual or group (beliefs about being danger among the paranoid?)

Type-III biases arise from our avoidable ignorance of facts or lack of insights, the possession of which would have improved our epistemic accuracy across a broad domain.  (Many of Eliezer’s recent postings appear to aim to overcome Type III bias, for example by explaining important facts about evolution, which would help us form more accurate belief about many specific issues that are illuminated by evolutionary biology.)  We distinguish subtype-III(a) resulting from lack of (procedural) insights about methodology, logic, or reasoning principles (e.g. anthropic bias), and subtype-III(b) resulting from lack of (substantial) knowledge about theoretical or concrete facts (e.g. errors resulting from ignorance about the basic findings of evolutionary psychology).

The distinctions between these different types are not always clear-cut.  For example, biases of Type I and II may often be overcome by the right kind of information: does not this mean that they are really Type III biases?  But I think in many cases we can reasonably judge what the principal source of the error is: just as when somebody comes down with pneumonia we can point to bacterial infection as the principal cause, not their failure to take a prophylactic dose of antibiotics, even though antibiotics would have prevented and may now be the way to overcome the disease.

Type III bias fades into simple error from unsystematic ignorance.  I think the most paradigmatic kind of bias is Type I bias.

If we consider the statements in addition to beliefs, we can add a fourth type of bias: misrepresentation.  This type of bias occurs when an individual or organization makes statements that systematically misrepresent its real beliefs.

We can further expand the concept so that it can be applied to objects that are neither beliefs nor statements.  We could say that a body of data is biased, for example, if the most straightforward interpretation of it gives a systematically misleading picture of reality.  Similarly, a scientific instrument could be biased if it tends to deliver biased data.

If so, is there an alternative explanation other than that they are biased?

(a) Maybe shareholders don’t want management to be able to blame poor results on bad luck due to fires etc?  But it seems relatively easy to check if accidents of this sort were responsible for big losses – easier, e.g., than to check whether competition has been unusually difficult.

(b) Maybe shareholders prefer to make certain pure bets:  whether company X has a good business plan, for example, rather than whether company X will also be lucky with regard to fires, floods, etc.

(c) Maybe the amounts spent on insurance are so small that shareholders just don’t care.  But in some cases I would think insurance costs are substantial, e.g. for shipping firms or transport of hazardous materials. Do such companies not buy insurance?

(d) Maybe costumers and suppliers want to know that the company will not suddenly go bankrupt because of a plant fire.  But I doubt that these constituencies ever check how much fire insurance a corporation has bought.

(e) Maybe when making long-term plans, it is valuable to reduce uncertainty as much as possible.  So a corporation might insure against big fires for the same reason that it hedges against currency risks.  But hedging currency risk may not be as expensive as insuring against plant accidents (given thick, efficient currency markets).  And where capital markets are sufficiently efficient, couldn’t a previously profitable firm with good prospects simply raise more capital if it unexpectedly needed to rebuild a burnt down factory?

Or is loss aversion bias partially responsible?

One might think that cognitive style a purely matter of taste, with no right or wrong.  Alternatively, one might think that different people have different strengths and weaknesses, and that it makes sense for individuals to adopt a cognitive style that makes the best use of the cognitive resources they have.  For example, somebody who is good at numbers should make more use of numerical data; while someone who is weak in math should employ more qualitative or narrative modes of cognition.  On this view, there is right and wrong, but it is relative: different for different people. 

A third alternative is that there is an optimal cognitive style that we should all attempt to approximate.  Through accidents of genetics and development, we diverge from this ideal in different directions.  But we can learn from others and from our own experience to calibrate our tendencies to better resemble the ideal human epistemic agent.  (A similar set of views could be formulated about emotional style, or personality.)

It seems to me that the true picture is a mixture of these three extreme alternatives.  Robin’s penchant for what Tyler calls “logical atomism” might be, in part, simply reflecting Robin’s subjective taste, in part a strategy to optimize the use of Robin’s talents, and in part a specific approximation to a universal human epistemic ideal – an approximation which perhaps Tyler believes would be improved if Robin became less “atomistic” (more like Tyler?).

When I think about how I ought to try to change myself, I can see that I am different in some respects from most other people. I think some of these differences are for the better, others for the worse. But often I don’t know which are which!

Perhaps I ought to divide the differences into the three classes described above. Then I could be happy to remain the way I am with regard to differences that are either a matter of pure taste, or a matter of optimizing the use of my idiosyncratic endowments. With regard to the third class, though, should I be as reluctant to remain unique as I should be (under certain conditions) about disagreeing with others about narrowly factual propositions? And in concrete terms, which of our personality traits fall into this third class?

I’ve noticed in some projects that I’ve been involved with that there were many steps each of which, at the time, were said to be and gave the appearance of being "the really crucial one, the one that would determine the success or failure of the project". Having passed one hurdle, there was another one — this time the really critical one. Then another, the really really critical one. Then one more…

Maybe project managers produce inflation in the currency of urgency. In order to maximize the effort of their teams, they hype each stage as being more important and urgent than it really is. Once the team catches on, the manager must increase the hype even more, just to achieve the same effect. In the end, every task must be a priority flag in order to get done at all.

I’m trying avoid doing this, but I suspect that I am thereby making my communication less effective when I’m talking to audiences whose "importance-meter" has been calibrated to speakers who routinely use emphatic language to get attention and to underscore the importance of what they are saying. 

Some entertainment makes use of a similar device to milk the audience for as much affect as possible. The hero faces a life-threatening situation, like a big monster, but survives – phew. Then an even bigger monster appears – surely this is the make-or-break challenge! Yes, he made it! But look out, behind his back is an enormous monster about to attack!

Another illustration (which seemed good at first) is offered by my email program, which has a feature whereby emails can be flagged as more or less urgent. There are five levels, which we might render as ‘very urgent’, ‘urgent’, ‘normal’, ‘less urgent’, and ‘not urgent’. I’ve hardly ever received an email with a priority level below ‘normal’. The merely ‘urgent’ flag is also almost never used. Practically all emails in which the sender has changed the default level away from ‘normal’ arrive as ‘very urgent’.

But this might not be a good example, since emails that are not at least at the ‘normal’ level should not be sent at all, and people might simply not bother to adjust the priority level unless the email is ‘very urgent’.

Nevertheless, I would speculate that there is a widespread "importance bias", which is in some ways analogous to, yet distinct from overconfidence bias, and also distinct from a general "entertainment" bias of futuristic prediction.

We might even be subjecting outselves to a kind of importance bias in the form of the documented durability bias in affective forecasting — we tend to overestimate how long we will experience joy or sadness in reaction to favorable or unfavorable life events. (But the explanation for why we have this bias might well be different from the explanation for why there is a general importance bias in some of our communication.)

One sign that science is not all bogus is that it enables us to do things, like go the moon. What practical things does debiassing enable us to do, other than refraining from buying lottery tickets?

In this context, it is not so helpful to adduce controversial philosophical or futuristic conclusions, such as that one should sign up for cryonics, reallocate all one’s charity to combat existential risk, or focus one’s work on creating Friendly AI. For presumably it would be as easy to delude oneself that these conclusions are correct as it is to delude oneself that one has been successful in overcoming bias and that one has thereby become an importantly better epistemic agent.

Consistent long-term success in active stock market speculation would be an impressive proof. But to require that would be to set the standard too high. Presumably, markets already suffer much less from bias than many other contexts, so even if one cannot beat the market one might nevertheless have gained some important ability.

But in what sphere of application does success at overcoming bias yield uncontroversial practical benefits?

Might it be that the only clear benefits of overcoming bias are "benefits" that most people don’t seek? Perhaps in contexts where people actually care about the outcome, and where bias significantly affects the outcome, they are already pretty unbiased? If so, it would seem that overcoming bias is more of a moral enterprise than anything else. Only if you happen to adopt a new and unusual goal will you benefit from putting work into overcoming biases (namely, those biases that previously were irrelevant but  now stand in the way of your effectively pursuing the new goal).

So one reason for asking (non-rhetorically) what overcoming bias is good for is to find ways of testing how successful we have been in increasing our epistemic effectiveness. Another reason is that if we knew in what domains being more skilled in unbiased thinking provides the greatest performance enhancements, we could shift our work more into such domains. For example, are there some kinds of academic work where being savvy about biases yields unusually large advantages? Is there an easy way to make money from being less biased?

If this new tool is really as great as we might like to think, then where are its profitable uses?

Also, the majority of people in the world are not atheist (or non-religious, or secular). Absent reasons to weight some opinions more, what should one believe when there are several inconsistent views, none of them commanding majority support?

I think in such a case one should believe a superposition of the views, i.e. one assigns a probability measure over the alternatives that reflects the degree of support they each have from their various constituencies. In the unrealistic, simplest case, where everyone’s reliability is the same and errors are uncorrelated, this might perhaps amount to assigning probability proportional to number of proponents.

Assuming the unrealistic simplifying premiss, in Hal’s case this would amount to being uncertain but not dismissive about spiritual matters, say being an agnostic who tends to believe that some existing religion is probably right, but not sure which one although more likely one of the big ones than some minor cult.

Of course, you might find that almost everybody would agree that such agnosticism is wrong, and you would find yourself in disagreement with this overwhelming majority. But it would nevertheless seem to be the position that would minimize disagreement.

A separate problem is what you should do if you end up with this belief. Suppose each religion claims that you will go to hell unless you believe that particular religion with all your heart. In that case, your rational course of action might be to pick the most likely religion and then do what you can to try to become a full convert to it.

The existence of such extreme disagreements as in the religious case, however, strongly suggests that not everybody involved is unbiased an in honest pursuit of objective truth. Some other factors must play a huge role in determining religious belief. So you might also think that by carefully examining what those non-rational factors are, you may be able to do better than minimizing disagreement; you might reach some insights that would make it rational for you to take sides. Of course, it is easy to delude oneself into thinking that one has such special insights, so one should be cautious. 

There are situations where even considering (deliberating about, giving conscious attention to) some possible course of action sends a bad signal. Sometimes taking that sort of action might be necessary. But ideally, you don’t want to be considering whether to take that action unless you really will take the action. In such cases, you may be uncomfortable with weak clues. If you notice and acknowledge such weak cues at all, they force you to consider whether they are strong enough that you need to take the dread action; yet they are likely too weak, so you won’t take the action after all. All you have done is to send out the bad signal.

Consider examples such as accusing an employee of stealing, a spouse of cheating, or declaring war on a neighboring country. Sending the signal that you are thinking about whether the weak clues you have are sufficient to warrant embarking on these drastic courses of action may well sour your relationships. In these cases, weak clues can be worse than useless.

Of what components would cognitive forensics be built? I’d think it would have a big chunk of applied statistics, but also contributions from cognitive and social psychology, epistemology, history and philosophy of science, sociology of science, maybe some economics, data mining, network analysis, etc.

Compared to this blog, the field could have somewhat narrower scope, focusing primarily on empirical scientific research rather than on rationality in general. It might also focus primarily on statistical tests rather than on wider issues such as institution design (although ideas for institutional reform might emerge as a side product). It might be driven more by statistical analysis of particular data sets than by big theories of common human cognitive biases (although the latter would serve as a source of inspiration for hypotheses to test).

The time might be ripe for this sort of endeavor. I have the impression that scattered articles on the problems of peer review and on possible statistical biases in scientific research (e.g. by funding source, file drawer effect etc.) are now appearing fairly regularly in Science and Nature.

Three questions I have are: (1) to what extent would it make sense to study *motivated* scientific error semi-separately (as a sub-discipline) rather than as part of the course of statistics and scientific methodology in general? (2) to what extent does such a sub-discipline already exist today? (3) if there is a need for a new sub-discipline, should it be as envisaged here or should it be constructed in a different way?