Why We Believe

Long ago, I first believed in religion as a young kid because I believed what I was told. Then I also believed because religious claims seemed to explain the strong emotions that religious contexts induced. This is how religion works – you feel strong emotions due to candles, buildings, clothes, music, well crafted and button-pushing words, charismatic empathetic leaders, social support and status. And if respected leaders and supporters around you then claim that your emotions are caused by God, well that makes sense. Even though many religious claims are transparently crazy, at least to people who well understand the world, they are easy for the young or inexpert to accept.

Recently while watching an emotional movie with political and moral overtones, I was reminded that the same is true for art. Art can make us feel strong emotions via all the same mechanisms. When high status artists and art supporters around us tell us these emotions are caused by our recognizing the emotional truth of art’s moral, political, and legal claims, that can make sense to us. Yet most of the channels by which art makes us feel emotions are irrelevant to the truth of its key claims. When we come to see this, we usually make excuses and tell ourselves that we aren’t fooled by all that other stuff; we really are just evaluating only the key moral/political/legal arguments. But the many correlations we see between features of art and who is persuaded when make it hard to believe this applies to most people most of the time.

The same likely also holds for essays like this one, or academic papers. While such writings may contain logical arguments, they also transmit writing styles, author charisma, status, and impressiveness, and clues about who supports or opposes them. You might think that you correct for all those influences when you read such writings and evaluate their claims, but the patterns above in religion and art suggest this is unlikely. The fact that people aren’t very interested in the accuracy of their pundits suggests we usually give a high priority to presentation style.

Could we do better? On subjects that have implications for future observations we could use prediction markets. But what about other subjects? Well, we might try to control for presentation variation by having a group of neutral writers rewrite common arguments in a standard style. That is, a single neutral writer could present all the different arguments on some subject, all using the same writing style. Readers of such presentations would have a better chance of drawing conclusions on each subject based on the logic of arguments, instead of writing styles. The fact that we aren’t very interested in these sort of presentations suggests that we aren’t very interested in reducing the influence of other writing style related factors.

GD Star Rating
loading...
Tagged as: , ,
Trackback URL:
  • Charlene Cobleigh Soreff

    _How_ would one rewrite the presentation in a neutral style? Even something a far from politics as the presentation of the analysis of a subroutine usually needs to put some emphasis on particular points in order to make the connections between the stages in the argument clear.

    • Ben

      I’d rather shoot for arguments to be rewritten in their strongest possible form, rather than in a “neutral style”.

      • http://overcomingbias.com RobinHanson

        If all options are all written in the same strongest form, that *is* a style that is neutral between the options.

      • Ben

        Fair enough. My mind went to the journalist’s definition of neutrality, which I think is fairly different. We want/expect the rewrites to favor the strongest ideas, which is either neutral or not depending on how you think about it.

      • http://don.geddis.org/ Don Geddis

        That sounds like how the pro/con arguments in official election publications are produced. Each side is allowed to present their best text, to try to convince the voters. For that matter, legal representation is supposed to work this way too: the job of a defense lawyer is NOT to objectively evaluate the evidence (that’s the job of the judge/jury), but instead to present the most compelling possible (legal) case for their side. The hope is that if both sides put forth their best effort, then the (strong!) non-logical influences will cancel out, and the only remaining differences will be objective truth. Doesn’t always work perfectly, but it does seem a reasonable approach to this dilemma.

      • IMASBA

        In politics two sides throwing dirt at each other doesn’t just cancel out the dirt. For example it matters a lot who threw a particular kind of dirt first. Court cases are somewhat less worse, but the same complications still play a role (especially in legal systems with jury trials). The moral of the story is that there are many, many biases in the human mind, Robin tackles some of them but he tackles them separately while in practice they will interact and efforts to reduce one kind of bias may increase another.

      • http://juridicalcoherence.blogspot.com/ Stephen Diamond

        What you’re missing is that—like other forms of social coordination— communication is hard.

        Hard enough that any style can be improved (if not necessarily by the original writer). Every intellectual writer purports to try to write in the “strongest form possible.” But in doing so, the writer can only exploit his own (modular) endowments, which are unique. Thus, each writer trying to produce the strongest possible version will write in an individualized style because he can only exploit his own strengths.

        To the extent that we’re attracted by particular modular configurations making up a particular writer, we’re exhibiting an irrational influence of style. But, if we’d like to eliminate this bias, we still wouldn’t have a single style, unless we do it by opting for a uniform mediocre one (with less than optimal expressive power).

        Therefore, unless everything is written by a single Great Writer (who is magically informed about all points of view), seeking the strongest form will not result in the same style.

      • http://juridicalcoherence.blogspot.com/ Stephen Diamond

        What you’re missing is that—like other forms of social coordination—communication is hard.

        Hard enough that any style can be improved (if not necessarily by the original writer). Every intellectual writer purports to try to write in the “strongest form possible.” But in doing so, the writer can only exploit his own (modular) endowments, which are unique. Thus, each writer trying to produce the strongest possible version will write in an individualized style because he can only exploit his own strengths.

        To the extent that we’re attracted by particular modular configurations making up a particular writer, we’re exhibiting an irrational influence of style. But, if we’d like to eliminate this bias, we still wouldn’t have a single style, unless we do it by opting for a uniform mediocre one (with less than optimal expressive power).

        [[In The profession’s disdain for “fine writing”: The sociology (the OB software won’t allow a link), I suggest that one function of legalese has been to supply a writing style that’s neutral and mediocre.]

        Therefore, unless everything is written by a single Great Writer (who is magically informed about all points of view), seeking the strongest form will not result in the same style.

      • http://overcomingbias.com RobinHanson

        I proposed having all the arguments on a single issue written by the same person. I didn’t say that single person had to write everything.

      • http://juridicalcoherence.blogspot.com/ Stephen Diamond

        True, I’m sorry that my “Great Writer” point seemed to allude to you proposal.

        Applying the reasoning specifically to your proposal, the null hypothesis should be that we choose versions based on the writer’s expressive power. If we prefer specific styles more than we prefer a single style with greater expressive power (the “stronger form”), this would accord with your hypothesis.

        In fact, we do sometimes prefer the stronger single writer to the original exponents. Isn’t this what textbooks in the social science present when they deal with controversial topics? There’s still a flourishing market for textbooks. (Despite lack of expressive power being problematic with many.)

        Why do you focus on presentations, rather than the far larger textbook market? Presentations, after all, are about personalities.

    • http://www.gwern.net/ gwern

      Is this a serious question or a rhetorical stylish way of saying you don’t think it’s a good idea? If it’s the former, here’s one way off the top of my head (and I’m sure there’s a great many other possibilities): reduce the arguments to a logic like propositional logic and present that. If that’s too hard for most people to understand, you can have someone translate it back into English.

      • http://juridicalcoherence.blogspot.com/ Stephen Diamond

        Leibnitiz had that idea, but we know now that natural language isn’t reducible to propositional logic.

        Moreover, even if you could, the “formalization” of broad ideas would itself be subject to style.

      • http://www.gwern.net/ gwern

        Language may not be entirely reducible to a logic, but that’s not necessarily important: style, threats, insults, status moves – all these may be key to natural languages, but exactly the sort of thing we *don’t* want to encode into our arguments (it’s not a bug, it’s a feature). Those are the things we wish to avoid driving our decision-making.

        > Moreover, even if you could, the “formalization” of broad ideas would itself be subject to style.

        The style would be orthogonal to the politics and policies, I think, and if one didn’t like a logical style, well, one can always rewrite the proofs: as long as they start with the same axioms and end in the same theorems…

      • http://juridicalcoherence.blogspot.com/ Stephen Diamond

        1. Let me be more precise, then: descriptive language isn’t completely reducible to propositional logic. (To my best knowledge of the state of the debate.)

        2. I don’t understand your other point: you can rewrite ordinary prose. (And prose style isn’t highly confounded with politics.)

        3. The basic point (how I construed Charlene’s) is that trying to equate styles can only succeed through loss of expressive power. (Therefore, our disinclination to seek a “neutral” style isn’t irrational insofar as it can be explained by seeking expressive power.)

      • http://www.gwern.net/ gwern

        > descriptive language isn’t completely reducible to propositional logic.

        Why not? Wouldn’t be just a lot of predicates and quantifiers?

        > I don’t understand your other point: you can rewrite ordinary prose…can only succeed through loss of expressive power.

        Of course you can… and that’s what’s under discussion here, the best way to rewrite ordinary prose to extract the semantic content which matters (models, evidence, predictions about the future, and cost-benefits of policies) while lossily omitting the other semantic content (dog whistles, rhetoric, style).

      • http://juridicalcoherence.blogspot.com/ Stephen Diamond

        On the first, you’re ignoring the huge tacit component in all communication.

        On the second … basically the same. Your (faulty) model is that information is “contained” in prose. Prose is a (at best, partly successful) means by which the writer tries to evoke the same ideas in his readers as he entertains. To truly “extract” the information contained in prose, you would have to read the writer’s mind.

      • arch1

        This is an interesting discussion you’re having w/ gwern. You might find it easier to sort out your different perspectives in the context of a specific example.

      • IMASBA

        “To truly “extract” the information contained in prose, you would have to read the writer’s mind.”

        I’m inclined to agree. Words like “much”, “very”, “a lot”, “dangerous”, etc… are all subjective and even worse they depend on the mood and recent experiences of the person and the person could not consciously tell you what definitions they effectively use or which formula describes their change under the influence of moods and recent experiences. This phenomenon seems very difficult for people to understand and/or accept and it has given rise to misconceptions such as the sorites “paradox” and an infinite number of pointless political and philosophical discussions on the “true” meaning of some word.

      • http://www.gwern.net/ gwern

        > On the first, you’re ignoring the huge tacit component in all communication.

        Hopelessly vague an objection.

        > Your (faulty) model is that information is “contained” in prose.

        Bizarre, I know – that symbols and data might have meaning? Surely we are too postmodern to believe such naivetes.

      • VV

        You mean something like Cyc? It seems pretty much like a failed attempt.

      • http://www.gwern.net/ gwern

        Cyc shows that a structured ontology is not what you really need for AI; however, it doesn’t show data or logic is useless – ‘the unreasonable effectiveness of data’ etc.

  • adrianratnapala

    You claim that “Essays like this one” pull the same tricks as emotional art in the sense that the author uses the most persuasive writing style possible. This is true but limited, the tone of this post, and similar articles either on the web or in real journals, is much more neutral and less emotional than any movie or novel. That tone is required in order to make it persuasive.

    I contend there is a competition between writers to make their essays seem as non-bullshit as possible and that it draws them towards some approximation of your “neutral writer”. I don’t claim this occurs in every corner of the web (or even academia), but I do think neutrality-of-style it is pretty common and includes writers with known political agendas.

  • John Hartman

    Signalling probably also counts. When you don’t have the time or effort to rigorously critically analyse everything you see, factors other than the argument come into account. Someone who writes eloquently is probably smart and educated, so he’s more likely to be right. An unusually charismatic leader is more likely to genuinely believe in his cause. A political ideology that creates fanaticism in your friends is more likely to be right, as more people agree with it.

    It’s obvious that emotion plays more of a role in decision making than it should, but it would be just impractical to logically assess the validity of everything that you come across.

    • IMASBA

      Indeed, in the AI/neuroscience world it is believed that emotions serve to focus a mind on a few things when there are too many things to logically analyze with a feasible amount of computational power. Emotions probably worked just fine for decision making when foragers still lived in small groups in a simpler world. In our modern world we can do some things to inoculate ourselves against the undesired effects of emotional arguments, but this requires vast resources if it is to be spread among as much of the population as possible, so don’t expect miracles and expect a lot of depressions and the like among those who would find their lives empty without strong emotional arguments to guide them.

      • Doug

        In machine learning there’s the concept of “bagging”. Many weak predictors trained in different ways often outperform a strong predictor that’s trained in the most optimal way. An ensemble of weak predictors has higher “bias”, but lower “variance”. Basically it’s less likely to get it exactly right, but more likely to avoid being very wrong.

        I suspect that emotion vs reasoning is a very similar dichotomy in human cognition. Particularly on fuzzy topics, precise logical reasoning can get way off base if only a few weak predicates are incorrectly assumed. Gut feeling rarely produces any deep insights, particularly counterintuitive ones, but it usually avoids leading to insane conclusions.

      • http://overcomingbias.com RobinHanson

        Yes, the brain is using many weak clues. But they are not so much clues about what is true as clues about what is socially useful to believe in your current context.

      • Doug

        I agree with that. But you seem to imply that this type of thinking is overall more prone to hypocrisy than logical reasoning. I’m not entirely convinced. On the one hand aggregating a weak clues in a diverse way produces conclusions that are much less tractable than reducing the problem to a strong and comprehensive model. It’s easier to obscure the truth by selectively ignoring certain weak clues, whereas logical reasoning can be audited for flaws.

        On the other hand it’s well known that weak but diverse ensembles are more robust to noise than strong but complex models. An adversary can inject just a little bit of bad information into just the right points in the model, and have it reach a very different conclusion.

        Gut feeling or common sense probably allows more hypocrisy on most issues, but it bounds the overall amount of hypocrisy on any given issue. Observationally it seems that the ideologies and belief systems that have the most absurd beliefs tend to rely more on logical gymnastics rather than base emotional appeal.

      • http://juridicalcoherence.blogspot.com/ Stephen Diamond

        You mean more robust to systematic noise, don’t you? (They would seem more sensitive to random noise.)

  • Dan

    When I read this essay, I thought it was leading up to Wikipedia’s Neutral Point of View.

    • http://effectivealtruismhub.com/user/tom-ash Tom Ash

      Yes, Wikipedia isn’t a *terrible* approximation of this suggestion: “a single neutral writer could present all the different arguments on some subject, all using the same writing style.”

  • IMASBA

    This remind me of films and books that use emotional narratives to explain the choices the protagonist makes or to show who the bad guys are. Usually it’s lazy writing that could just as well have been used to show the protagonist or his faction being the bad guys.

    On the issue of opposing views being written by the same person, so with the same style. This already happens in journalism and is actually a major problem because humans are also wired to take the amount of arguments into account, this causes the problem/bias of false balance in articles about science and politics.

  • arch1

    Really good post and (so far at least:-) quality comments, thanks. The 2nd paragraph reminds me of G. H. Hardy’s “A Mathematician’s Apology” in which he quotes Shakespeare’s “Not all the waters in the rough rude sea can wash the balm off an anointed king” and asks whether anything could be more beautiful and less true.
    Improvements in this regard seem very important though (as IMASBA observes) they would be swimming upstream against our evolved brain structure etc.
    Somewhat related to gwern’s idea, I recall seeing from time to time mentions of knowledge representation systems intended to capture the structure of evolving debates as a way of facilititaing consensus/convergence on truth, but I’m not aware of anything that has gained traction. Does anyone know more on this topic?

  • Pingback: Recomendaciones | intelib

  • brendan_r

    “Could we do better?”

    I think we used to. Pop-sci from Morris Kline and George Gamow, stuff written in the 1930-1960 period, is written more plainly than any popular science today. Is math beautiful? I’m sure Kline thought so, but he didn’t insult the reader by making it the theme of his book.

    I love Neal Stephenson, but I can’t be his only reader that suspects I love him because his stuff is so impressively (intentionally?) hard to understand that I pat myself on the back for occasionally understanding him. Pournelle wrote some stuff that was technically difficult for me, but I never get the impression of injecting difficulty (obscure vocab) for impressiveness’ sake.

    And of course there is TED, which is the embodiment of ego stroking fluff over substance.

  • Pingback: Sentences to Ponder | relinquishment

  • Pingback: Twitter’s science stars

  • Michael Caton

    To echo John Hartman a bit – evaluating arguments entirely on their own merits is difficult. Instead, mostly we rely on a heuristic approach, which is to weight the argument based on clues about the holder of the arguments (which style and status are part of). The quality of the arguer is obviously not identical to the quality of the argument, but it’s not totally unrelated. Then again this approach is profoundly vulnerable to truth-evaluation-clouding tribalism.

    Why do we do this? It could just be we’re using hardware that’s not designed for impartial argument evaluation; we don’t usually evaluate arguments accurately just because we can’t. There’s also the not-mutually-exclusive possibility that ***most of the things we read on a daily basis are entertainment, and we
    don’t really expect them to affect a substantial consequential decision we’re going to make at some point.*** This second possibility is most disturbing to those of us who spend our time on blogs signalling our intellects as a big part of our concept of self-worth. It’s also related to the first possibility, in the sense that until very recently, humans didn’t have access to powerful enough tools to make our beliefs actually matter much in terms of decisions we make in the world; there was no inclusive fitness value to such a talent, and since it’s unclear that it affects inclusive fitness even today, there probably still is not.

  • Keith

    What you suggest at the end seems like a version of the Delphi method.

    https://en.wikipedia.org/wiki/Delphi_method

  • Ivan

    The problem is that I think people are looking for that emotional element and are very unlikely to seek out something written in a very neutral style

  • http://www.DanDzombak.com/ Dan Dzombak

    “Well, we might try to control for presentation variation by having a
    group of neutral writers rewrite common arguments in a standard style. That is, a single neutral writer could present all the different
    arguments on some subject, all using the same writing style. Readers of such presentations would have a better chance of drawing conclusions on each subject based on the logic of arguments, instead of writing styles.”

    I’ve found that the magazine “The Week” does this very well, with one writer presenting all the different opinions on the big news stories of the past week.

  • Pingback: Overcoming Bias : Forged By Status