Even if you're familiar with the ideas that are presented on this blog, it can be surprising just how strong the forces of habit and rote cognition and behavior can be.

One of the schools of cognitive psychology that addresses biases describes "system 1" and "system 2" thinking, where "system 1" is everyday automatic processing, deciding by intuition, relying on heuristics, and totally filled with biases, and "system 2" is thoughtful and careful consideration, logical and methodical. But this seems inadequate, because we can slip into automatic cognitive patterns even when we are consciously trying to be careful.

A few examples from personal experience below the fold…

I experienced this directly when playing the gatekeeper in an AI-box experiment arranged with another OB reader. I agreed not to give the details of what happened, but I think I can say this much without violating that agreement: I found my own mind responding to the person playing the AI as if he were actually an AI. Even though I knew it was a person who was playing the role of an AI, and even though I knew it was perfectly permissible to completely disregard everything my partner/opponent was saying and recite Blake poetry at him, something in the fact of the assigned roles — some kind of automatic suspension of disbelief — caused me to think as if there was actually an AI on the other end of the internet. I still won, but it was surprisingly difficult.  I think that surprising difficulty came about in part because I responded in that way: I took what my partner was saying <i>seriously</i>, even though there was no need to do so. 

Similarly, I was recently on the receiving end of a bizarre bit of rote behavior. Someone did something obnoxious, I caught him and called him out, and he tried to lie his way out of it with the classic phrase "I have no idea what you're talking about." I'm pretty sure nobody has ever uttered that phrase honestly — it's like something that's culturally encoded in us: someone calls us out on something, we activate the Outraged Denial Subroutine, and the phrase "I have no idea what you're talking about" flies out of our mouths (or out of an e-mail, which took some time to write, from an intelligent person!). (Dear readers: ask yourself whether you have ever used that phrase sincerely.) But do we believe it by rote too?

A third example: I play a lot of chess, particularly three-minute speed chess on FICS. At that speed, there's no time really to calculate long lines of moves, there's just time to play tactical swipes that you see (for non chess-players, a tactic is a short series of moves that are meant to achieve some kind of advantage, usually material — like attacking two of your opponent's pieces at once). There's also time to follow intuitions: to move pieces into positions that you know are strategically good, or play intuitive attacks (sacrificing material when the enemy king looks vulnerable, for example). Chess is a favorite game for studying bounded rationality and the ways that skilled versus unskilled players think — Herbert Simon influentially thought that the difference between grandmasters and ordinary patzers mostly was in the number of "chunks" — of memorized positions — accessible to immediate cognition. But it seems like there's more, in this sense. I notice in three-minute chess that I have tactically good periods and bad periods, even thought my overall (low) level of skill is presumably the same throughout, and even though I am trying just as hard. Sometimes, I play by pure intuition, and sometimes I have a calculation capacity available to me and can perform tactics, and there's no visible difference between those times, except that I lose a lot in the first case and win a lot in the second.  Chess players talk a lot about stamina, and this might be what they mean: the mental discipline to play well might just run out after a while.

This is all very strange, and it makes me wonder if we can think properly even if we try. How much of rationality — of being a good Bayesian Ninja or whatever — isn't about intelligence, or knowing how to think, but about having the self-control and discipline to exercise those capacities? And what does it mean for our attempts to become more rational if, as a lot of recent psychological research has been suggesting, our self-control generally is a limited resource?

How can we overcome rote cognition, if it sticks around even when we're trying our best to be mentally alert and careful?