When Does Evidence Win?

Consider a random area of intellectual inquiry, and a random intellectual who enters this area. When this person first arrives, a few different points of view seemed worthy of consideration in this area. This person then becomes expert enough to favor one of these views. Then over the following years and decades the intellectual world comes to more strongly favor one of these views, relative to the others. My key question is: in what situations do such earlier arrivals, on average, tend to approve of this newly favored position?

Now there will be many cases where favoring a point helps people to be seen an intellectual of a certain standing. For example, jumping on an intellectual fashion could help one to better publish, and then get tenure. So if we look at tenured professors, we might well see that they tended to favor new fashions. To exclude this effect, I want to apply whatever standard is used to pick intellectuals before they choose their view on this area.

There will also be an effect whereby intellectuals move their work to focus on new areas even if they don’t actually think they are favored by the weight of evidence. (By “evidence” here I also mean to include relevant intellectual arguments.) So I don’t want to rely on the areas where people work to judge which areas they favor. I instead need something more like a survey that directly asks intellectuals which views they honestly think are favored by the weight of evidence. And I need this survey to be private enough for respondents to not fear retribution or disapproval for expressed views. (And I also want them to be intellectually honest in this situation.)

Once we are focused on people who were already intellectuals of some standing when they choose their views in an area, and on their answers to a private enough survey, I want to further distinguish between areas where relevant strong and clear evidence did or did not arrive. Strong evidence favors one of the views substantially, and clear evidence can be judged and understood by intellectuals at the margins of the field, such as those in neighboring fields or with less intellectual standing. These can included students, reporters, grant givers, and referees.

In my personal observation, when strong and clear evidence arrives, the weight of opinion does tend to move toward the views favored by this evidence. And early arrivals to the field also tend to approve. Yes many such intellectuals will continue to favor their initial views because the rise of other views tends to cut the perceived value of their contributions. But averaging over people with different views, on net opinion moves to favor the view that evidence favors.

However, the effectiveness of our intellectual world depends greatly on what happens in the other case, where relevant evidence is not clear and strong. Instead, evidence is weak, so that one must weigh many small pieces of evidence, and evidence is complex, requiring much local expertise to judge and understand. If even in this case early arrivals to a field tend to approve of new favored opinions, that (weakly) suggests that opinion is in fact moved by the information embodied in this evidence, even when it is weak and complex. But if not, that fact (weakly) suggests that opinion moves are mostly due to many other random factors, such as new political coalitions within related fields.

While I’ve outlined how one might do a such a survey, I have not actually done it. Even so, over the years I have formed opinions on areas where my opinions did not much influence my standing as an intellectual, and where strong and clear evidence has not yet arrived. Unfortunately, in those areas I have not seen much of a correlation between the views I see as favored on net by weak and complex evidence, and the views that have since become more popular. Sometimes fashion favors my views, and sometimes not.

In fact, most who choose newly fashionable views seem unaware of the contrary arguments against those views and for other views. Advocates for new views usually don’t mention them and few potential converts ask for them. Instead what matters most is: how plausible does the evidence for a view offered by its advocates seem to those who know little about the area. I see far more advertising than debate.

This suggests that most intellectual progress should be attributed to the arrival of strong and clear evidence. Other changes in intellectual opinion are plausibly due to a random walk in the space of other random factors. As a result, I have prioritized my search for strong and clear evidence on interesting questions. And I’m much less interested than I once was in weighing the many weak and complex pieces of evidence in other areas. Even if I can trust myself to judge such evidence honestly, I have little faith in my ability to persuade the world to agree.

Yes if you weigh such weak and complex evidence, you might come to a conclusion, argue for it, and find a world that increasingly agrees with you. And you might let your self then believe that you are in a part of the intellectual world with real and useful intellectual progress, progress to which you have contributed. Which would feel nice. But you should consider the possibility that this progress is illusory. Maybe for real progress, you need to instead chip away at hard problems, via strong and clear evidence.

GD Star Rating
Tagged as: , ,
Trackback URL:
  • Lord

    How much of the path to strong and clear evidence winds through weak and complex evidence? Or is it more important to focus on incomplete and insufficient evidence? Isn’t weak and complex evidence just the deficiency of the pursuit of incomplete and insufficient evidence to lead anywhere? Foxes and hedgehogs.

    • http://overcomingbias.com RobinHanson

      Yes of course most strong and clear evidence arose out of more weak and complex evidence. But we can make a great many useful distinctions among the latter in terms of its chances of leading to the former.

  • Anon.

    Surely capitalism plays a big role. A wise man once said:

    “There cannot be science without an effective social mechanism for the elimination of failure, based on extra-rational criteria, inaccessible to cultural capture.”

  • Michael Vassar

    Or maybe chipping away at hard problems anti-correlates with searching for strong and clear evidence.

  • http://juridicalcoherence.blogspot.com/ Stephen Diamond

    As a result, I have prioritized my search for strong and clear evidence on interesting questions.

    Robin, you’ve recently complained about publication delay. Why not cite to it (as in publication) to provide examples? I wonder what “strong and clear evidence” you have in mind.

    [On the subject of what near-mode evidence produces far-mode belief change: see “Cognitive Dissonance: The Glue of the Mind” ( http://juridicalcoherence.blogspot.com/2017/02/270-cognitive-dissonance-glue-of-mind.html ) It is the beginning of a series that seeks to synthesize cognitive dissonance theory, construal level theory, and reactance theory.]

  • Sid

    Could you give a few examples? It’s bit too abstract right now…

  • http://www.iamasciencelady.co.uk I Am A Science Lady

    It’s interesting that the question is framed in terms of convincing fellow intellectuals of an argument through evidence. Those people will have had, if not training in formal logic, training in evaluating information and understanding the strength of an argument. The majority of the population has not had an education that prepares them in this way, and so arguments that are popular and appeal to what makes people “feel good” tend to win out (examples: Brexit, Trump).
    In the context of a community of intellectuals, the net movement of opinion is towards the newest facts and findings. But what would this look like in the general population? We know that changes in the outlook of a society do happen (e.g. 1960s & 70s sexual revolution), but is the rate of change slower? I don’t know if there’s existing research on this (I’m an engineer by trade), and I would love to learn more.

  • Silent Cal

    Weighing the first, abstract part of this post, I concluded that if early arrivals don’t approve of later trends, then either the trends aren’t caused by the evidence, or the early arrivals’ approval isn’t.

    I’ll take for what it’s worth your inside view that the intellectual world doesn’t respond to weak or complex evidence. But the first half of this post is an attempt to set up an outside view, and you seem to elide the fact that on that outside view it could just as easily be you who is wrong.

  • http://juridicalcoherence.blogspot.com/ Stephen Diamond

    Was the em book based on weak clues? If so, does your reappraisal of weak clues speak to some misdirection? What evidence led you to reject an emphasis on weak clues? (Were those clues weak or strong?) There’s so much that might be addressed!

    About four years ago, I argued against a weak-clue orientation. http://disq.us/p/f9s2om

  • https://dndebattbetyg.wordpress.com/ Stefan Schubert

    Good post. Another reason to focus on strong evidence is that it probably correlates with *strong effects* in many cases. For instance, it is normally easier to find strong evidence that one intervention is more effective than another if the difference in terms of effects is large. But it also matters more then.