Disagreement Case Studies

I’d love to hear posts (on this blog or others) describing disagreement case studies.  That is, tell us about a specific disagreement (on a matter of fact not value) that you have with specific other reasonably respectable people, and tell us how you reconcile that disagreement with the irrationality of forseeing to disagree

You know their opinion, and they know your opinion, and yet you hold differing opinions.   You realize that both your opinion and theirs may result in part from defects, such as thinking errors or not knowing something that the other knows.  So:

  • Do you conclude just from the fact that they disagree that they must have more defects?
  • Do you think they realize that they can have defects, such as thinking errors or knowing less?
  • Should the fact that you disagree be a clue to them about their defects?   Is it a clue about yours? 
  • Do they adjust their estimates enough for the possibility of their defects?  If not, why not?
  • What clues suggest to you that they have more defects, or under-adjust for them?
  • What clues suggest to them that you have more defects, or under-adjust?
  • Do you both have access to these clues, and if so do you interpret them differently? 
  • Do you each realize some clues might be hidden?   
  • Does your inability to answer any of these questions suggest you have defects?
  • Consider all these questions again for your meta-disagreement about who has more defects.

Some people’s answers come down to "I just know I’m smarter; I have no reasons." I am writing a book on disagreement and want to include case studies like yours in my book.

Added:  If many are involved in your disagreement, consider these questions  about other people on all sides. If your case is interesting, I’m willing to interview you by phone to walk you through this line of questioning. 

GD Star Rating
Tagged as:
Trackback URL:
  • http://profile.typekey.com/sentience/ Eliezer Yudkowsky

    “The best part about math is that, if you have the right answer and someone disagrees with you, it really is because they’re stupid.”
    Quotes from Honors Linear Algebra

  • http://pdf23ds.net pdf23ds

    Well, I guess this doesn’t directly answer your question, but it’s related. I think it’s possible that, to some extent, we may persist in defending a position when talking to someone even after adjusting our probability assignments based on the evidence of them holding their position. This is rational, even though actually holding the position you’re defending would not be, because arguing about it could lead to uncovering new information that would make the issue clearer for both people.

    So when I’m arguing politics with someone to the right of me, I shift to the left, and vice versa. But that doesn’t mean my opinions are actually changing in the meanwhile.

  • http://profile.typekey.com/robinhanson/ Robin Hanson

    Eliezer, disagreeing with the right answer on any subject is a weak sign of stupidity; I’m not sure why it would be a stronger sign in math than any other subject.

    Pdf, I seek case studies of actual disagreement, not where people only pretend to disagree.

  • http://profile.typekey.com/halfinney/ Hal Finney

    I have cases where I disagree with groups of people. These are usually areas of controversy where there are more people on the other side. That is, I take the side of the majority or the “conventional wisdom” as I interact and debate with groups that vigorously advocate minority views.

    It’s difficult to have a simple, personal, one-on-one disagreement with someone unless the topic is a factual matter that is private to the two of you. Otherwise you are both essentially representatives of a larger dispute that may have large numbers of people and arguments on both sides. Now you are still personally disagreeing about which of these large sets of arguments is more persuasive, so there is still a personal disagreement involved. But at the same time, you can’t move towards agreement with the other person without coming into disagreement with a much larger group. It’s not clear what the right thing is to do in this situation.

    For case studies, perhaps you would do best with localized factual disagreements, although they seem rather unimportant.

  • http://profile.typekey.com/robinhanson/ Robin Hanson

    Hal, you are right that analysis can be cleaner in cases of a few people who know each other. When more people are involved, then I’d really want people to answer these questions about both sides of the dispute.

  • Doug S.

    You’re probably sick of my Magic: The Gathering references, but last year there were a group of renowned pro players who all moved into a house together to try to decide on what deck (strategy) to play. All but one of the players decided to play the same deck. The lone dissenter won the tournament and took home the top prize of $40,000.

    Official magicthegathering.com event coverage can be found here.

    Again, read this for the best analysis I’ve ever seen as to when disagreement is helpful or not helpful.

  • michael vassar

    Couldn’t it also be that on a particular issue they are less concerned about overcoming the biases that you are worried about; possibly because they are more concerned about overcoming other possible biases?

    For instance, biases towards overconfidence and biases towards accepting consensus not because it is likely to be true but because doing so is convenient, low risk, or accompanied by approval, often tend to direct behaviors in opposite directions.

  • http://profile.typekey.com/sentience/ Eliezer Yudkowsky

    In the vast majority of cases, I attribute my persistent disagreements to asymmetrical clue possession.

  • http://profile.typekey.com/jdannan/ James Annan

    My recent/ongoing disagreement on estimation of climate sensitivity is well-documented on my blog. I justify my disagreement on the basis that I can identify substantial errors in the existing literature (including the prosecutor’s fallacy, and/or the misguided belief that a uniform prior actually encapsulates pure ignorance, combined with misconceptions about the nature of the problem we are trying to solve in the first place), and even understand why they have arisen. Basically, many people started out with frequentist hypothesis-testing – P(D|H) statements – but really want Bayesian probability statements – P(H|D). There is enough vague verbiage in climate science that many don’t appreciate the difference, and the political pressure is such that critical evaluation is limited (we have to be a consensus against the sceptics).

    As for my opponents, I think the main reason they’ve sustained their position so dogmatically is common-or-garden denial, bolstered by the belief that the majority agree with them (probably true, but it’s a shrinking one), therefore they don’t need to take me seriously.

  • http://pdf23ds.net pdf23ds

    Robin, I suspect that people tend to act like this without being completely aware of it. That is to say, they don’t give other people as much weight as would be required by Aumann’s theorem, but they give others more weight that they’re aware of letting on to. They might become aware if put into a situation where they have to make an important decision based on their position, or maybe even not then. (And this is notwithstanding any other biases, some of which have opposite effects in many circumstances.)

  • http://profile.typekey.com/robinhanson/ Robin Hanson

    James, many people document their disagreements in the sense of listing arguments for their position and against other positions. I’m asking instead for a more meta description, addressing the kind of questions I listed.

    Eliezer, similarly, I seek more details; what sort of clues?

    pdf, all the more reason to look at cases where we ask people to reflect on their disagreement.

    Michael, I’m not sure I follow you.

  • http://profile.typekey.com/tschoegl/ Adrian Tschoegl

    I take it we have ruled out matters of religion or taste (literally, such as my avoidance of offal) as not being amenable to reason.

  • michael vassar

    Robin: I am saying that we have many known biases, and often different biases tend to push in opposite directions. In particular, there are often biases pushing towards and away from accepting the consensus within some group. It is therefore possible that people who are relatively more concerned with avoiding one bias than another will differ in their acceptance of group consensus, will be able to justify their disagreement with those relatively more concerned with avoiding some other bias.

    An obvious example is that overconfidence creates bias towards undervaluing consensus, but that social pressures very often create bias towards accepting consensus, or at least towards accepting the consensus among one’s social peers (few people are tempted to accept the flat earth consensus of most of the humans who have ever lived).

    People who wish to submit fully to overconfidence can and do justify it in terms of resisting biases towards conformity. In their favor, they have the observation that in the absence of overconfidence information cascades should generally be expected to cause a strong group consensus to be little more informative than one random group member’s opinion. Overconfidence reduces the magnitude of this effect.

    People who wish to submit fully to the biases favored by conformity can justify it in terms of efforts to overcome overconfidence.

  • http://fashion-incubator.com/mt Kathleen Fasanella

    I invited someone to join my blog as a co-author specifically because we disagree. I know that I can’t be immune to errors (and she points them out) and neither is she. Rather, we each represent inconsistent thinking in our respective camps; our errors are shared by those who respectively agree with each of us. I feel that in order to provide the greater service to our readership, that this is the best way to point out inconsistencies marring our perceptions and actions. I also tend to be opinionated and stubborn but in spite of that, I want readers to do what’s right and that often means not following my lead. Some of our debates are so “spirited” that visitors have often wondered why I have her on there or suppose that we dislike each other. The fact is, we are probably best friends (among friends who’ve never met in real life) and although she is much younger and less experienced than me, I admire her greatly.

    Similarly, when we *do* agree, I feel those arguments have greater weight, of which there is no ambiguity which lends itself as an example others would do well to follow (ours is an advice/educational blog).

  • Stuart Armstrong

    Getting people to change their opinion is not that hard, I’ve found; getting them to admit that their opinion has changed is very hard, though. And if you point it out to them, they often revert to their original position.

    Maybe because we’re brought up to see discussion as a contest, with winners or losers. We aim to gain wisdom through the discussion, but not to “lose” by having to admit that we are wrong. Even those who feel socratic, who feel that truth can emerge from discusions, may be unwilling to “give up”, to not give their own side “a fair hearing”.

    On some deeply held opinions, we may be unwilling to disagree with our past selves – “I can’t remember exactly why I believed this at some point in the past, but I did, and the reasons must have been convincing, even if I can’t recall them to mind immediately”.

    To answer some of Robin’s original questions:
    I’m much more skilled and analytical than most people in conversation; from my analytical ability I conclude that people who end up disagreeing with me are more likely to have deficiencies in their analysis. From my skill, however, I conclude that even if people do end up agreeing with me, that doesn’t mean I was right.

  • http://profile.typekey.com/robinhanson/ Robin Hanson

    Michael, I think you are saying that disagreement about some particular topics can be traced to disagreements about which sorts of biases are how severe. Similarly, many disagreements seem traceable to disagreements about which person is smarter/wiser.

  • http://felixsalmon.com Felix Salmon

    My answers to your questions here.

  • http://profile.typekey.com/jdannan/ James Annan

    OK, in more detail:

    # Do you conclude just from the fact that they disagree that they must have more defects?

    No, I conclude it based on an evaluation of the case for and against, including the reaction of others when I have the opportunity to present my case to them.

    # Do you think they realize that they can have defects, such as thinking errors or knowing less?

    I really don’t know what is going on in their heads (and there are several of them, so they may have a range of opinions).

    # Should the fact that you disagree be a clue to them about their defects? Is it a clue about yours?


    # Do they adjust their estimates enough for the possibility of their defects? If not, why not?

    I guess they judge themselves to be more expert than me, and also they believe that most people accept their side of things.

    # What clues suggest to you that they have more defects, or under-adjust for them?

    I know (from personal discussion) that many of their supposed “supporters” don’t actually support them explicitly from a position of understanding, but merely as a default of trusting people they consider to be experts.

    # What clues suggest to them that you have more defects, or under-adjust?

    The fact that they’ve got away with it for about 5 years with no-one pulling them up on it.

    # Do you both have access to these clues, and if so do you interpret them differently?

    Yes, I interpret this acquiescence as being due to the fact that most people (in climate science) haven’t thought about it very carefully, instead automatically trusting those they consider to be experts. (Most of them are basically physicists who don’t claim to know much about probability).

    # Do you each realize some clues might be hidden?

    We are both unaware of each others’ private conversations. They might not realise quite how many people I have presented my ideas to.

    # Does your inability to answer any of these questions suggest you have defects?

    Fortunately I managed to answer them all, which means I’m perfect 🙂

    # Consider all these questions again for your meta-disagreement about who has more defects.

    Oh, please…

  • http://profile.typekey.com/halfinney/ Hal Finney

    I see some interesting contrasts among the disagreements described by Felix, James and me. Felix and I find ourselves in a similar position: our respective disputants are relatively expert compared to us, and have strong and loyal followings. At the same time I think we are both advocating a generally mainstream view, while our disputants are mavericks in their fields, whose positions are in the minority. One difference is that Felix really thinks his arguments are superior, while I am more inclined to view quality of argument as not a very useful signifier of truth.

    James is in something of the opposite position, acting as a lonely voice trying to persuade the larger mass of opinion to his view. Although it seems to have been an uphill battle for him, he persists, confident in the superior quality of his arguments.

    From the “meta” perspective, the odds are against James. Most people who find themselves in positions like his turn out to be mistaken. Nevertheless, as a scientist, he perhaps has a professional obligation to advocate and defend his views so that they get a fair hearing. Just as we don’t want a legal system where lawyers make their own judgments about guilt and then will only argue the side they think is right, society would probably not be benefited by a scientific community whose members were too quick to adopt the majority view, even though that would increase the average accuracy of individual scientists.

  • rcriii

    I wrote up a dispute and my answers to the questions here.

  • http://profile.typekey.com/robinhanson/ Robin Hanson

    Rciii, thanks for trying this! Your disagreement seems to a version of the classic “inside view” (detailed analysis) vs. “outside view” (experience with similar cases). See Kahneman, D. and Lovallo, D. (1993) Timid choices and bold forecasts. A cognitive perspective on risk taking. Management Science, 39, 17-31.

    Felix, thanks also! I’ll leave my comments on your blog.

  • http://profile.typekey.com/robinhanson/ Robin Hanson

    James, you say most people defer to a few people who oppose you, thinking those few are more expert. You focus your comments on these many followers, but don’t say much about these few they follow. What clues suggest their defects? Are they not in fact more expert?

  • http://profile.typekey.com/jdannan/ James Annan


    Assuming that truncated comment was simply “are they not in fact more expert” then I am sure that most people would consider them to be. However, it’s worth noting that this is a rather new area of climate science and although they moved sideways into it before I did, they didn’t have any real background in it before that time. Neither did anyone else in climate science, which is why they were able to establish themselves as experts without much meaningful review.

    One clue that suggests their defects to me is that they have not been able to coherently and clearly state their case.

    Another is that there are plenty of “meta-experts” who are far more expert in the field of probabilistic estimation in general – but not climate science in particular – who have been completely scathing about what has been going on in the field. In no small part they are frozen out by the established climate science clique.

  • http://profile.typekey.com/robinhanson/ Robin Hanson

    James, we should talk more about how to determine the “real” experts on topics.

  • http://profile.typekey.com/jdannan/ James Annan

    I don’t think a truth-seeker can ever simply trust an expert qua expert to give them “the answer”. At least, not a truth-seeker who is close enough to the coal-face (and sufficiently motivated) to actually look into the question themselves (it’s probably a reasonable heuristic in practical cases). But each new question is never quite the same as any preceding one, and expertise is specialised and limited and based on what went before…

  • rcriii

    Robin, That is a great paper, lots to absorb.

    Yesterday I put on my ‘outsider’ hat and looked at the effect of the configurations we were arguing about on past performance, and found little data to support any particular answer. So should I throw up my hands and let the PM dictate configuration, or should I dictate my preferred answer (I still have all that theory, see)? Or is the rational thing to stop wasting time on analysis and flip a coin?

  • Kirk Shanahan

    The first step is always to define what you’re talking about. In the case I am commenting on, I disagree that there exists a body of scientific information that establishes with a high probability that a nuclear process known as ‘cold fusion’ exists which produces ‘excess heat’. To that end, I formulated and published what I consider to be a non-nuclear explanation for the extant body of data regarding apparent excess heat. That publication has been challeged directly once, and I have defended it. And it has been indirectly challenged another time, and I defended that (both times in print). So:

    # Do you conclude just from the fact that they disagree that they must have more defects?

    # Do you think they realize that they can have defects, such as thinking errors or knowing less?
    Yes, in principle, but in practice they believe they have eliminated all significant probability that they have errors. (I believe they also believe a non-expert in the field could not identify significant defects.)

    # Should the fact that you disagree be a clue to them about their defects? Is it a clue about yours?
    Normally, the fact that a disagreement has been published is both an indication that I disagee, which should be a clue to them, and it also indicates at least one other agrees with me (via the peer review system), which should also be a clue. The fact that the one I am disagreeing with also has published an intended rebuttal indicates the same for that side, which should be a clue to me I suppose. Both sides supposedly try to publish the clearest presentation of their position they can.

    # Do they adjust their estimates enough for the possibility of their defects? If not, why not?
    This is a little hard to answer, but I would say no, because as I have
    noted before, my counterproposal is ignored, rather than acted on. In
    science when a valid counterproposal is made, meaning one cannot find
    a valid reason to reject it, the standard response is to go back to the
    lab and conduct new and different experiments aimed at resolving the
    issues. Why they don’t do this would be pure speculation on my part.
    I have indicated that the one author I have dealt with in detail
    seems to just simply reject my proposal for nonscientific reasons.

    # What clues suggest to you that they have more defects, or under-adjust for them?
    That their published proposal does not consider at all my
    counter-proposal. That when they are in receipt of my
    counter-proposal they reject it for spurious reasons.

    # What clues suggest to them that you have more defects, or under-adjust?
    Good question. The short answer is that I don’t know. They have
    suggested that I am incapable of cogent commenting unless I have
    personally conducted equivalent experiments, which I have not.
    However, my proposal rests on data analysis methods, not data
    collection methods, so their point is moot. But this may not
    be the primary reason they do not accept my proposal.

    # Do you both have access to these clues, and if so do you interpret them differently?
    Absolutely yes to both questions. That is the crux of the issue.
    We propose different mechanisms that produce the ‘clues’ (data).

    # Do you each realize some clues might be hidden?
    Of course, finding the hidden clues (getting the data) is what science is all about. A scientific publication, while supposedly presenting a very good case, is actually a request to the general audience for any relevant comments. Theoretically, the ‘deal is not ever done’, and the best theories are subject to rejection. In practice, a lot of disagreements arise from people refusing to act like this is true.

    # Does your inability to answer any of these questions suggest you have defects?
    I wasn’t unable to answer for myself. I also don’t have ESP, so
    I can’t say for the others. Is that a defect?

    # Consider all these questions again for your meta-disagreement about who has more defects.
    OK, I take this question to mean I should consider why my ‘opponents’
    won’t listen to me, and vice versa. The problem is that I have listened
    to them, in great detail. If my counter-proposal is not correct, their
    position is the default fall-back position, which I have never denied
    (in other words, cold fusion is a possibility if no other explanation
    works). On the other side, they don’t accept my position, but are unable
    to technically defend their lack of acceptance, which is the requirement
    for doing ‘good’ science, i.e. if they want to reject my proposal without
    supplying new relevant data, they have to define a logical error.
    which they have not done.

    So does this help for your case study?

  • http://profile.typekey.com/robinhanson/ Robin Hanson

    Kirk, this is helpful yes, thank you.