I’d love to hear posts (on this blog or others) describing disagreement case studies. That is, tell us about a specific disagreement (on a matter of fact not value) that you have with specific other reasonably respectable people, and tell us how you reconcile that disagreement with the irrationality of
The first step is always to define what you're talking about. In the case I am commenting on, I disagree that there exists a body of scientific information that establishes with a high probability that a nuclear process known as 'cold fusion' exists which produces 'excess heat'. To that end, I formulated and published what I consider to be a non-nuclear explanation for the extant body of data regarding apparent excess heat. That publication has been challeged directly once, and I have defended it. And it has been indirectly challenged another time, and I defended that (both times in print). So:
# Do you conclude just from the fact that they disagree that they must have more defects?No.
# Do you think they realize that they can have defects, such as thinking errors or knowing less?Yes, in principle, but in practice they believe they have eliminated all significant probability that they have errors. (I believe they also believe a non-expert in the field could not identify significant defects.)
# Should the fact that you disagree be a clue to them about their defects? Is it a clue about yours?Normally, the fact that a disagreement has been published is both an indication that I disagee, which should be a clue to them, and it also indicates at least one other agrees with me (via the peer review system), which should also be a clue. The fact that the one I am disagreeing with also has published an intended rebuttal indicates the same for that side, which should be a clue to me I suppose. Both sides supposedly try to publish the clearest presentation of their position they can.
# Do they adjust their estimates enough for the possibility of their defects? If not, why not?This is a little hard to answer, but I would say no, because as I havenoted before, my counterproposal is ignored, rather than acted on. Inscience when a valid counterproposal is made, meaning one cannot finda valid reason to reject it, the standard response is to go back to thelab and conduct new and different experiments aimed at resolving theissues. Why they don't do this would be pure speculation on my part.I have indicated that the one author I have dealt with in detailseems to just simply reject my proposal for nonscientific reasons.
# What clues suggest to you that they have more defects, or under-adjust for them?That their published proposal does not consider at all mycounter-proposal. That when they are in receipt of mycounter-proposal they reject it for spurious reasons.
# What clues suggest to them that you have more defects, or under-adjust?Good question. The short answer is that I don't know. They havesuggested that I am incapable of cogent commenting unless I havepersonally conducted equivalent experiments, which I have not.However, my proposal rests on data analysis methods, not datacollection methods, so their point is moot. But this may notbe the primary reason they do not accept my proposal.
# Do you both have access to these clues, and if so do you interpret them differently?Absolutely yes to both questions. That is the crux of the issue.We propose different mechanisms that produce the 'clues' (data).
# Do you each realize some clues might be hidden?Of course, finding the hidden clues (getting the data) is what science is all about. A scientific publication, while supposedly presenting a very good case, is actually a request to the general audience for any relevant comments. Theoretically, the 'deal is not ever done', and the best theories are subject to rejection. In practice, a lot of disagreements arise from people refusing to act like this is true.
# Does your inability to answer any of these questions suggest you have defects?I wasn't unable to answer for myself. I also don't have ESP, soI can't say for the others. Is that a defect?
# Consider all these questions again for your meta-disagreement about who has more defects.OK, I take this question to mean I should consider why my 'opponents'won't listen to me, and vice versa. The problem is that I have listenedto them, in great detail. If my counter-proposal is not correct, theirposition is the default fall-back position, which I have never denied(in other words, cold fusion is a possibility if no other explanationworks). On the other side, they don't accept my position, but are unableto technically defend their lack of acceptance, which is the requirementfor doing 'good' science, i.e. if they want to reject my proposal withoutsupplying new relevant data, they have to define a logical error.which they have not done.
Yesterday I put on my 'outsider' hat and looked at the effect of the configurations we were arguing about on past performance, and found little data to support any particular answer. So should I throw up my hands and let the PM dictate configuration, or should I dictate my preferred answer (I still have all that theory, see)? Or is the rational thing to stop wasting time on analysis and flip a coin?
I don't think a truth-seeker can ever simply trust an expert qua expert to give them "the answer". At least, not a truth-seeker who is close enough to the coal-face (and sufficiently motivated) to actually look into the question themselves (it's probably a reasonable heuristic in practical cases). But each new question is never quite the same as any preceding one, and expertise is specialised and limited and based on what went before...
Assuming that truncated comment was simply "are they not in fact more expert" then I am sure that most people would consider them to be. However, it's worth noting that this is a rather new area of climate science and although they moved sideways into it before I did, they didn't have any real background in it before that time. Neither did anyone else in climate science, which is why they were able to establish themselves as experts without much meaningful review.
One clue that suggests their defects to me is that they have not been able to coherently and clearly state their case.
Another is that there are plenty of "meta-experts" who are far more expert in the field of probabilistic estimation in general - but not climate science in particular - who have been completely scathing about what has been going on in the field. In no small part they are frozen out by the established climate science clique.
James, you say most people defer to a few people who oppose you, thinking those few are more expert. You focus your comments on these many followers, but don't say much about these few they follow. What clues suggest their defects? Are they not in fact more expert?
Rciii, thanks for trying this! Your disagreement seems to a version of the classic "inside view" (detailed analysis) vs. "outside view" (experience with similar cases). See Kahneman, D. and Lovallo, D. (1993) Timid choices and bold forecasts. A cognitive perspective on risk taking. Management Science, 39, 17-31.
Felix, thanks also! I'll leave my comments on your blog.
I see some interesting contrasts among the disagreements described by Felix, James and me. Felix and I find ourselves in a similar position: our respective disputants are relatively expert compared to us, and have strong and loyal followings. At the same time I think we are both advocating a generally mainstream view, while our disputants are mavericks in their fields, whose positions are in the minority. One difference is that Felix really thinks his arguments are superior, while I am more inclined to view quality of argument as not a very useful signifier of truth.
James is in something of the opposite position, acting as a lonely voice trying to persuade the larger mass of opinion to his view. Although it seems to have been an uphill battle for him, he persists, confident in the superior quality of his arguments.
From the "meta" perspective, the odds are against James. Most people who find themselves in positions like his turn out to be mistaken. Nevertheless, as a scientist, he perhaps has a professional obligation to advocate and defend his views so that they get a fair hearing. Just as we don't want a legal system where lawyers make their own judgments about guilt and then will only argue the side they think is right, society would probably not be benefited by a scientific community whose members were too quick to adopt the majority view, even though that would increase the average accuracy of individual scientists.
# Do you conclude just from the fact that they disagree that they must have more defects?
No, I conclude it based on an evaluation of the case for and against, including the reaction of others when I have the opportunity to present my case to them.
# Do you think they realize that they can have defects, such as thinking errors or knowing less?
I really don't know what is going on in their heads (and there are several of them, so they may have a range of opinions).
# Should the fact that you disagree be a clue to them about their defects? Is it a clue about yours?
Obviously
# Do they adjust their estimates enough for the possibility of their defects? If not, why not?
I guess they judge themselves to be more expert than me, and also they believe that most people accept their side of things.
# What clues suggest to you that they have more defects, or under-adjust for them?
I know (from personal discussion) that many of their supposed "supporters" don't actually support them explicitly from a position of understanding, but merely as a default of trusting people they consider to be experts.
# What clues suggest to them that you have more defects, or under-adjust?
The fact that they've got away with it for about 5 years with no-one pulling them up on it.
# Do you both have access to these clues, and if so do you interpret them differently?
Yes, I interpret this acquiescence as being due to the fact that most people (in climate science) haven't thought about it very carefully, instead automatically trusting those they consider to be experts. (Most of them are basically physicists who don't claim to know much about probability).
# Do you each realize some clues might be hidden?
We are both unaware of each others' private conversations. They might not realise quite how many people I have presented my ideas to.
# Does your inability to answer any of these questions suggest you have defects?
Fortunately I managed to answer them all, which means I'm perfect :-)
# Consider all these questions again for your meta-disagreement about who has more defects.
Michael, I think you are saying that disagreement about some particular topics can be traced to disagreements about which sorts of biases are how severe. Similarly, many disagreements seem traceable to disagreements about which person is smarter/wiser.
Getting people to change their opinion is not that hard, I've found; getting them to admit that their opinion has changed is very hard, though. And if you point it out to them, they often revert to their original position.
Maybe because we're brought up to see discussion as a contest, with winners or losers. We aim to gain wisdom through the discussion, but not to "lose" by having to admit that we are wrong. Even those who feel socratic, who feel that truth can emerge from discusions, may be unwilling to "give up", to not give their own side "a fair hearing".
On some deeply held opinions, we may be unwilling to disagree with our past selves - "I can't remember exactly why I believed this at some point in the past, but I did, and the reasons must have been convincing, even if I can't recall them to mind immediately".
To answer some of Robin's original questions:I'm much more skilled and analytical than most people in conversation; from my analytical ability I conclude that people who end up disagreeing with me are more likely to have deficiencies in their analysis. From my skill, however, I conclude that even if people do end up agreeing with me, that doesn't mean I was right.
I invited someone to join my blog as a co-author specifically because we disagree. I know that I can't be immune to errors (and she points them out) and neither is she. Rather, we each represent inconsistent thinking in our respective camps; our errors are shared by those who respectively agree with each of us. I feel that in order to provide the greater service to our readership, that this is the best way to point out inconsistencies marring our perceptions and actions. I also tend to be opinionated and stubborn but in spite of that, I want readers to do what's right and that often means not following my lead. Some of our debates are so "spirited" that visitors have often wondered why I have her on there or suppose that we dislike each other. The fact is, we are probably best friends (among friends who've never met in real life) and although she is much younger and less experienced than me, I admire her greatly.
Similarly, when we *do* agree, I feel those arguments have greater weight, of which there is no ambiguity which lends itself as an example others would do well to follow (ours is an advice/educational blog).
Robin: I am saying that we have many known biases, and often different biases tend to push in opposite directions. In particular, there are often biases pushing towards and away from accepting the consensus within some group. It is therefore possible that people who are relatively more concerned with avoiding one bias than another will differ in their acceptance of group consensus, will be able to justify their disagreement with those relatively more concerned with avoiding some other bias.
An obvious example is that overconfidence creates bias towards undervaluing consensus, but that social pressures very often create bias towards accepting consensus, or at least towards accepting the consensus among one's social peers (few people are tempted to accept the flat earth consensus of most of the humans who have ever lived).
People who wish to submit fully to overconfidence can and do justify it in terms of resisting biases towards conformity. In their favor, they have the observation that in the absence of overconfidence information cascades should generally be expected to cause a strong group consensus to be little more informative than one random group member's opinion. Overconfidence reduces the magnitude of this effect.
People who wish to submit fully to the biases favored by conformity can justify it in terms of efforts to overcome overconfidence.
Kirk, this is helpful yes, thank you.
The first step is always to define what you're talking about. In the case I am commenting on, I disagree that there exists a body of scientific information that establishes with a high probability that a nuclear process known as 'cold fusion' exists which produces 'excess heat'. To that end, I formulated and published what I consider to be a non-nuclear explanation for the extant body of data regarding apparent excess heat. That publication has been challeged directly once, and I have defended it. And it has been indirectly challenged another time, and I defended that (both times in print). So:
# Do you conclude just from the fact that they disagree that they must have more defects?No.
# Do you think they realize that they can have defects, such as thinking errors or knowing less?Yes, in principle, but in practice they believe they have eliminated all significant probability that they have errors. (I believe they also believe a non-expert in the field could not identify significant defects.)
# Should the fact that you disagree be a clue to them about their defects? Is it a clue about yours?Normally, the fact that a disagreement has been published is both an indication that I disagee, which should be a clue to them, and it also indicates at least one other agrees with me (via the peer review system), which should also be a clue. The fact that the one I am disagreeing with also has published an intended rebuttal indicates the same for that side, which should be a clue to me I suppose. Both sides supposedly try to publish the clearest presentation of their position they can.
# Do they adjust their estimates enough for the possibility of their defects? If not, why not?This is a little hard to answer, but I would say no, because as I havenoted before, my counterproposal is ignored, rather than acted on. Inscience when a valid counterproposal is made, meaning one cannot finda valid reason to reject it, the standard response is to go back to thelab and conduct new and different experiments aimed at resolving theissues. Why they don't do this would be pure speculation on my part.I have indicated that the one author I have dealt with in detailseems to just simply reject my proposal for nonscientific reasons.
# What clues suggest to you that they have more defects, or under-adjust for them?That their published proposal does not consider at all mycounter-proposal. That when they are in receipt of mycounter-proposal they reject it for spurious reasons.
# What clues suggest to them that you have more defects, or under-adjust?Good question. The short answer is that I don't know. They havesuggested that I am incapable of cogent commenting unless I havepersonally conducted equivalent experiments, which I have not.However, my proposal rests on data analysis methods, not datacollection methods, so their point is moot. But this may notbe the primary reason they do not accept my proposal.
# Do you both have access to these clues, and if so do you interpret them differently?Absolutely yes to both questions. That is the crux of the issue.We propose different mechanisms that produce the 'clues' (data).
# Do you each realize some clues might be hidden?Of course, finding the hidden clues (getting the data) is what science is all about. A scientific publication, while supposedly presenting a very good case, is actually a request to the general audience for any relevant comments. Theoretically, the 'deal is not ever done', and the best theories are subject to rejection. In practice, a lot of disagreements arise from people refusing to act like this is true.
# Does your inability to answer any of these questions suggest you have defects?I wasn't unable to answer for myself. I also don't have ESP, soI can't say for the others. Is that a defect?
# Consider all these questions again for your meta-disagreement about who has more defects.OK, I take this question to mean I should consider why my 'opponents'won't listen to me, and vice versa. The problem is that I have listenedto them, in great detail. If my counter-proposal is not correct, theirposition is the default fall-back position, which I have never denied(in other words, cold fusion is a possibility if no other explanationworks). On the other side, they don't accept my position, but are unableto technically defend their lack of acceptance, which is the requirementfor doing 'good' science, i.e. if they want to reject my proposal withoutsupplying new relevant data, they have to define a logical error.which they have not done.
So does this help for your case study?
Robin, That is a great paper, lots to absorb.
Yesterday I put on my 'outsider' hat and looked at the effect of the configurations we were arguing about on past performance, and found little data to support any particular answer. So should I throw up my hands and let the PM dictate configuration, or should I dictate my preferred answer (I still have all that theory, see)? Or is the rational thing to stop wasting time on analysis and flip a coin?
I don't think a truth-seeker can ever simply trust an expert qua expert to give them "the answer". At least, not a truth-seeker who is close enough to the coal-face (and sufficiently motivated) to actually look into the question themselves (it's probably a reasonable heuristic in practical cases). But each new question is never quite the same as any preceding one, and expertise is specialised and limited and based on what went before...
James, we should talk more about how to determine the "real" experts on topics.
Robin,
Assuming that truncated comment was simply "are they not in fact more expert" then I am sure that most people would consider them to be. However, it's worth noting that this is a rather new area of climate science and although they moved sideways into it before I did, they didn't have any real background in it before that time. Neither did anyone else in climate science, which is why they were able to establish themselves as experts without much meaningful review.
One clue that suggests their defects to me is that they have not been able to coherently and clearly state their case.
Another is that there are plenty of "meta-experts" who are far more expert in the field of probabilistic estimation in general - but not climate science in particular - who have been completely scathing about what has been going on in the field. In no small part they are frozen out by the established climate science clique.
James, you say most people defer to a few people who oppose you, thinking those few are more expert. You focus your comments on these many followers, but don't say much about these few they follow. What clues suggest their defects? Are they not in fact more expert?
Rciii, thanks for trying this! Your disagreement seems to a version of the classic "inside view" (detailed analysis) vs. "outside view" (experience with similar cases). See Kahneman, D. and Lovallo, D. (1993) Timid choices and bold forecasts. A cognitive perspective on risk taking. Management Science, 39, 17-31.
Felix, thanks also! I'll leave my comments on your blog.
I wrote up a dispute and my answers to the questions here.
I see some interesting contrasts among the disagreements described by Felix, James and me. Felix and I find ourselves in a similar position: our respective disputants are relatively expert compared to us, and have strong and loyal followings. At the same time I think we are both advocating a generally mainstream view, while our disputants are mavericks in their fields, whose positions are in the minority. One difference is that Felix really thinks his arguments are superior, while I am more inclined to view quality of argument as not a very useful signifier of truth.
James is in something of the opposite position, acting as a lonely voice trying to persuade the larger mass of opinion to his view. Although it seems to have been an uphill battle for him, he persists, confident in the superior quality of his arguments.
From the "meta" perspective, the odds are against James. Most people who find themselves in positions like his turn out to be mistaken. Nevertheless, as a scientist, he perhaps has a professional obligation to advocate and defend his views so that they get a fair hearing. Just as we don't want a legal system where lawyers make their own judgments about guilt and then will only argue the side they think is right, society would probably not be benefited by a scientific community whose members were too quick to adopt the majority view, even though that would increase the average accuracy of individual scientists.
OK, in more detail:
# Do you conclude just from the fact that they disagree that they must have more defects?
No, I conclude it based on an evaluation of the case for and against, including the reaction of others when I have the opportunity to present my case to them.
# Do you think they realize that they can have defects, such as thinking errors or knowing less?
I really don't know what is going on in their heads (and there are several of them, so they may have a range of opinions).
# Should the fact that you disagree be a clue to them about their defects? Is it a clue about yours?
Obviously
# Do they adjust their estimates enough for the possibility of their defects? If not, why not?
I guess they judge themselves to be more expert than me, and also they believe that most people accept their side of things.
# What clues suggest to you that they have more defects, or under-adjust for them?
I know (from personal discussion) that many of their supposed "supporters" don't actually support them explicitly from a position of understanding, but merely as a default of trusting people they consider to be experts.
# What clues suggest to them that you have more defects, or under-adjust?
The fact that they've got away with it for about 5 years with no-one pulling them up on it.
# Do you both have access to these clues, and if so do you interpret them differently?
Yes, I interpret this acquiescence as being due to the fact that most people (in climate science) haven't thought about it very carefully, instead automatically trusting those they consider to be experts. (Most of them are basically physicists who don't claim to know much about probability).
# Do you each realize some clues might be hidden?
We are both unaware of each others' private conversations. They might not realise quite how many people I have presented my ideas to.
# Does your inability to answer any of these questions suggest you have defects?
Fortunately I managed to answer them all, which means I'm perfect :-)
# Consider all these questions again for your meta-disagreement about who has more defects.
Oh, please...
My answers to your questions here.
Michael, I think you are saying that disagreement about some particular topics can be traced to disagreements about which sorts of biases are how severe. Similarly, many disagreements seem traceable to disagreements about which person is smarter/wiser.
Getting people to change their opinion is not that hard, I've found; getting them to admit that their opinion has changed is very hard, though. And if you point it out to them, they often revert to their original position.
Maybe because we're brought up to see discussion as a contest, with winners or losers. We aim to gain wisdom through the discussion, but not to "lose" by having to admit that we are wrong. Even those who feel socratic, who feel that truth can emerge from discusions, may be unwilling to "give up", to not give their own side "a fair hearing".
On some deeply held opinions, we may be unwilling to disagree with our past selves - "I can't remember exactly why I believed this at some point in the past, but I did, and the reasons must have been convincing, even if I can't recall them to mind immediately".
To answer some of Robin's original questions:I'm much more skilled and analytical than most people in conversation; from my analytical ability I conclude that people who end up disagreeing with me are more likely to have deficiencies in their analysis. From my skill, however, I conclude that even if people do end up agreeing with me, that doesn't mean I was right.
I invited someone to join my blog as a co-author specifically because we disagree. I know that I can't be immune to errors (and she points them out) and neither is she. Rather, we each represent inconsistent thinking in our respective camps; our errors are shared by those who respectively agree with each of us. I feel that in order to provide the greater service to our readership, that this is the best way to point out inconsistencies marring our perceptions and actions. I also tend to be opinionated and stubborn but in spite of that, I want readers to do what's right and that often means not following my lead. Some of our debates are so "spirited" that visitors have often wondered why I have her on there or suppose that we dislike each other. The fact is, we are probably best friends (among friends who've never met in real life) and although she is much younger and less experienced than me, I admire her greatly.
Similarly, when we *do* agree, I feel those arguments have greater weight, of which there is no ambiguity which lends itself as an example others would do well to follow (ours is an advice/educational blog).
Robin: I am saying that we have many known biases, and often different biases tend to push in opposite directions. In particular, there are often biases pushing towards and away from accepting the consensus within some group. It is therefore possible that people who are relatively more concerned with avoiding one bias than another will differ in their acceptance of group consensus, will be able to justify their disagreement with those relatively more concerned with avoiding some other bias.
An obvious example is that overconfidence creates bias towards undervaluing consensus, but that social pressures very often create bias towards accepting consensus, or at least towards accepting the consensus among one's social peers (few people are tempted to accept the flat earth consensus of most of the humans who have ever lived).
People who wish to submit fully to overconfidence can and do justify it in terms of resisting biases towards conformity. In their favor, they have the observation that in the absence of overconfidence information cascades should generally be expected to cause a strong group consensus to be little more informative than one random group member's opinion. Overconfidence reduces the magnitude of this effect.
People who wish to submit fully to the biases favored by conformity can justify it in terms of efforts to overcome overconfidence.