The cognitive architecture of bias

You’ll often hear people, including people on this blog, talk about self-deception as a central factor in creating bias.  Work from cognitive neuroscience, however, shows that the "self-deception" description doesn’t really capture how deep the issue runs.  In this post, I’ll give an analogy that a number of us use these days to describe our understanding of the cognitive architecture of conscious speech.  The earliest I can remember using the analogy was in a paper I wrote in the mid/late 90s, and the model behind it drives some of my comments over at Chatty Apes.  Jon Haidt gives the analogy in an interview here.  Rob Kurzban has a superb new article on it here.  Below, I provide the analogy.

Imagine some bureaucratic organization with a number of different departments.  A sort of caricature of George W. Bush’s presidential administration works.  There are a number of strong-willed people in key White House positions (including Bush, Dick Cheney, and Karl Rove) with overlapping but somewhat different agendas that lay out simple priorities (Cut taxes for rich people!  Get Saddam!  Help out our fat-cat cronies in defense and oil industries!  Keep us and people like us in power!).  There are a number of different topic-specific departments (Defense, Commerce, Education, etc.) with dedicated staffs that work on detailed operations.  Some departments or staffs are focused on coordinating action with other institutions like Congress or foreign governments.  And there are communications offices tasked specifically with coordinating information flow to the public in a way that will best allow the deep, unpublicized preferences of the Administration to be translated into action.

Inside the Administration, there is a constant balancing act between what they really want to do and what is possible.  Some things they may want to do are limited by their ability to get people (Congress, foreign governments, the public) to go along with it.  Here, communications strategies become crucial.  Maybe they want to cut taxes specifically for rich people, but they understand that most people aren’t rich and will strongly disapprove of a bunch of rich-only tax cuts.  The solution may be to combine very large tax cuts for rich people with very small tax cuts for middle-class people in a terrifically confusing package, and then tell everyone, consistently and earnestly, that one’s proposal simply involves "across the board" tax cuts for everyone.  If most people don’t have the information necessary to learn the truth, and mostly they’re just relying on the Administration’s communications to figure out whether it’s a good idea for them, the Administration could achieve both goals of cutting rich people’s taxes and being well thought of by middle-class voters.

So, the Administration decides to cut rich people’s taxes, and it comes up with a communication strategy that is maximally bent on effecting the policies and has little direct relationship to the real motivations behind the policies.  Optimally, the Administration will appoint press agents to communicate the message, and, even more optimally, will in fact not explain to the press agents what the real motives are behind the favored actions.  It will just tell those press agents the popular version with the popular motives.  The best liars believe the lie.

If you’re in a position to ask an Administration press agent about the policy, that person will tell you, consistently and earnestly, that throughout the Administration the motives are the unobjectionable ones.  The President thinks the fair thing is to give all hard-working Americans a little across-the-board tax relief.

Press secretaries often get it wrong through no fault of their own even with less vague things than motives.  Consider the Valerie Plame CIA leak scandal.  Early on, Scott McClellan, Bush’s press secretary at the time, positively insisted that Karl Rove had nothing to do with it.  "The President knows that Karl Rove wasn’t involved."  But then it turned out, not so surprisingly, that Rove and others were more than a little involved.  A likely story here is that Rove lied to McClellan and his staff, and that McClellan was sent out with bad information to be the most aggressive spokesman he could be.

Now the very difficult point:  This analogy is meant to be about the human brain — it’s your brain I’m talking about.  Research from psychology and neuroscience shows that your brain has organizational characteristics similar to this caricature of the Bush Administration.   There are systems that are responsible for powerful, simple emotional reactions that serve to focus other systems to give fine-tuned attention to your brain’s priorities and preferred outcomes.  Most of the detailed systems don’t care much about why they have the jobs they are given; they just do the work of carrying them out in a highly distributed, bureaucratic way.  And you — the conscious, chatty you — are that dimwitted patsy, the misinformed press agent.   The conscious you is not the President and you’re certainly not Karl Rove.  You are the Scott McClellan for your bureaucratic brain.  You’re constantly being duped into believing public-friendly stories about yourself, because your entire job is to tell stories handed to you by ruthless, clever, unconscious communications systems.  You’re not the whole head; you’re just the talking head.

You can’t even trust your own introspection, because your press agent simply doesn’t have direct access to the Oval Office.  The best you can do, even with your own behavior, is to try to piece together hypotheses about the hidden motives at work based on what the person actually does, situated in the context of things you know are generally true of why people might want to do those things.

"Self-deception" doesn’t fully capture the depth of what’s going on here.  When you talk to someone, you’re not talking to an integrated "self" that may or may not be deceived — you’re talking to a press agency that doesn’t really make many important decisions, that is only loosely connected to the rest of the bureaucracy, and that has the core mission not of truth-seeking, but bias that serves the interests of its genes.

The thing I really worry about:  To the extent that our scientific accounts of people’s underlying motivations end up painting, as I think they will, an unflattering picture about human predictability and genetic selfishness, I think we’re likely to end up with a small group of scientists who acknowledge it, and a very large group of people (including most scientists) with press agencies that are strongly motivated to deny it and so drift farther away from paying attention to the better work in social science.  How’s that for optimism?

GD Star Rating
Tagged as:
Trackback URL:
  • Robin Hanson

    I agree that we can usually think of our conscious minds as the gullible press agents of our minds. But then the question arises, how much can the press agent do to change the mind. At this point we might think of ourselves as the heroic rebel press agent, doing what he can to make the organization be more honest before he gets fired.

    In moments like these it might be more useful to think of yourself as the gullible summer intern working at the press office, who is being interviewed for a documentary on the press office. You have been told by our press office boss that the press office is fighting hard to be skeptical about what the rest of the organization tells it, and to get them to be more honest. And you the gullible summer intern, are so proud to work for such a courageous boss. Alas, your boss has lied to you.

  • Hal Finney

    That’s a great story, Weeden, and I’m sure there’s much truth to it. At the same time there seems to be a discrepency. The press agent does not believe that he runs the White House, yet we do believe that our conscious minds run our brains. It would be rather difficult to give Scott McClellan the impression that he ran things, yet somehow we are able to believe that about ourselves. So I tend to think that there is something missing in this analogy.

  • Jason Weeden

    Robin, I’m suspicious of talk that says, you know, sure *most* people are biased, but *I’m* really just trying to get at the truth. Isn’t that just what a good press agent would say? On the other hand, press agents do care about maintaining credibility, and there’s the key, I think. We can construct, monitor, and enforce rules of credibility with each other and try to force honesty out of inherently dishonest systems. This is what science strives to do, though often does a terrible job at it, precisely because scientists so often warp the rules of science to fit their own selfish and tribal ends. But it’s the best we’ve got.

    Hal, sure, it’s an incomplete analogy. In fact, our neural press agencies do have as part of their story that they run the whole show. But it’s not true. Look at the research from cognitive neuroscience and psychology — e.g., Mike Gazzaniga’s work on split-brains or Dan Wegner’s work on the illusion of conscious will — and you’ll find great examples. The fact that our press agents believe, falsely, that they run the show makes self-serving biases even harder to spot and correct than they are when we clearly recognize that we’re talking to a powerless spokesperson.

  • William Snowmaker

    Actually with the right kind of introspection the press-agent believed to be “you” can be seen right through.

    That is the core of Zen and other approaches to end belief in the self illusion. The self is simply a persistant thought-pattern, acquired around 18 months, refined through childhood and adolescence. This self illusion is responsible for the experience of mental suffering. This is not to imply that the self illusion disappears, but its heaviness and “center of gravity” tendency evaporate once it is seen not to be real.

  • Scoop

    Two questions:
    1. How much does this vary from person to person? Looking at my friends and their success with New Year’s resolutions, I know several people who seem pretty thoroughly controlled by their conscious selves and a lot more who are driven by unconscious forces.

    2. If things are so bleak, isn’t this site and nearly all attempt at introspection a waste of time? Should I abandon Overcoming Bias for porn?

  • Jason Weeden

    Scoop, on your first question, there’s a big psychological literature on self-control, ego depletion, etc., that says, yes, there are both differences between individuals in various contexts as well as differences within individuals from time to time and circumstance to circumstance. The deeper question the current research tries to get at it what it means to have *self* control or *ego* depletion in the first place. Kurban’s paper, which I referenced at the beginning of the post, has a discussion on this deeper point on page 7, so I’ll just point you there.

    I don’t know about bleak, but I’d definitely say that overcoming bias is really hard, harder than most people think. The fact that it’s so hard is an argument in favor of a project like Overcoming Bias, not an argument against it. If it were easy, we wouldn’t have to keep thinking about it and working at it.

  • TGGP

    The you vs your brain idea is convincingly argued against at but this post was pretty good as well.

    Nice save with the “caricature” note, this is a really high-quality blog and it would be very upsetting if it diverged too much into politics, a realm whose bull is among the most potent fertilizers of irrationality.

  • Nick Bostrom

    So you think of yourself as a gullible summer intern in the McClellan press office? Illusions of grandeur, my friend. In fact, you are all like mutated E. coli bacteria on the foot of a senile louse on the head of a gullible summer intern in the McCellan press office.

    Take that!

  • Robin Hanson

    Nick, :), I was limiting myself to creatures who had beliefs and statements, as we are “sure” of that part of our illusion. So maybe we are five year old kids pretending to be press office summer interns …

  • Brock

    Donald Davidson argued (e.g. in the last four essays in Problems of Rationality) that accounting for self-deception and similar forms of irrationality, such as weakness of the will, requires that the agent be divided into just such a “bureaucracy”.

    So rather than your model for cognitive bias being an alternative model to self-deception, I’d say that what your model is a re-statement of the Davidsonian model of self-deception.

  • John T. Kennedy

    “To the extent that our scientific accounts of people’s underlying motivations end up painting, as I think they will, an unflattering picture…”

    Why unflattering?

    If your evaluation of your selfish nature is negative, perhaps the problem is with that evaluation rather than with that nature.