Echo Chamber Confidence

We should realize that we gain far less info in an echo chamber than from being around folks with diverse views.  The latest Journal of Experimental Psychology says we just don't get this

The experimental task involved estimating the number of calories in measured quantities of different foods (e.g., a cup of yogurt, a bowl of cooked rice). … Participants were asked to generate a calorie estimate for each food and then indicate their confidence in it. … [Then] they were provided with the opinions of three advisors, and were given the opportunity to revise their initial estimates. They were told that they would receive a bonus for making accurate judgments, … [and] were also asked to indicate their confidence in their final (revised) estimates and to bet on their accuracy.  …

On half the trials (independent condition) the [screen] header stated that “these estimates were randomly drawn from a pool of 100 estimates made by participants in a previous study,” whereas on the remaining trials (opinion-dependent condition) the header stated that “these estimates were selected from those closest to your own initial opinion in a pool of 100 estimates made by participants in a previous study.” …

Receiving advice increased participants’ confidence in the dependent condition, but not in the independent condition.  Participants indicated greater confidence in their final estimates in the opinion-dependent than in the independent condition.  In accord with the confidence results, the participants bet more often in the dependent (58%) than in the independent condition (42%).

Please, please, don't let yourself succumb to the very common bias to confidence in views because "everyone" at your favorite website agrees with them, if those people have been selected for this very agreement!  Once you realize that many others elsewhere disagree, that disagreement should weigh heavily on your estimation. 

GD Star Rating
loading...
Tagged as: ,
Trackback URL:
  • http://lightskyland.com Matthew C.

    This seems extraordinarily relevant to OB on topics like AI and Cryonics, for example.

  • michael vassar

    What if everyone at our favorite website agrees with the above post but everyone elsewhere disagrees?

  • JustBored

    There’s surely strong disagreement on topics posted here, Matthew, but most don’t read comments and certainly don’t post them. As on all blogs, people prefer to lurk.

  • http://blog.moertel.com/ Tom Moertel

    Let’s assume that the participants in the study didn’t understand either of the two possible qualifications about the advice they were given. (Do most people really understand statements such as “these estimates were selected from those closest to your own initial opinion in a pool of 100 estimates made by participants in a previous study”?) That is, let’s assume that the qualifications meant *nothing* the participants beyond “here’s what three other people thought”. Wouldn’t we expect, then, to see results very much like what were reported in the study?

    Perhaps the findings don’t suggest that people discount interdependence as much as they suggest that people aren’t good at interpreting descriptions of sampling protocols.

  • Cyan

    To expand on Tom Moertel’s point, it seems like a plausible explanation for the increased confidence might simply be the closeness of the opinions to the original estimate, and people mostly don’t know how to deal with the information specifying how the opinions are selected. An improved experimental protocol might be to give half of those told that “these estimates were randomly drawn from a pool of 100 estimates made by participants in a previous study” opinions which were actually selected from those closest to their own initial opinions.

    None of the above militates against the advice at the end of the post.

  • James D. Miller

    For this reason, when I realized that I share with Eliezer a strong “pro-free market ideology” I became suspicious of my agreement with much of what he has written on the singularity.

  • Demosthenes

    “these estimates were selected from those closest to your own initial opinion in a pool of 100 estimates made by participants in a previous study.”

    I’m not sure this is even a “bias” anymore. It seems like this would suggest that people willfully believe that this is the best way to make decision contra

    “these estimates were randomly drawn from a pool of 100 estimates made by participants in a previous study,”

    Is this because they had a bias that has been repeatedly confirmed by confirmation bias? Is what started out as a flawed belief now the reigning theory?

  • Fenn

    So that’s where the bottom of the barrel went.

  • http://youlikeadajuice.wordpress.com/ Ben

    Methinks the key is not to avoid echo chambers (such as this one) but rather to embrace outsiders. Find intelligent people who disagree with you and engage them on ideas. You’re allowed to be snobby by using the intelligence qualifier, just recognize that intelligence isn’t the same as agreeing with you.

    Matthew C: of course, some of us are here looking for opposing views on AI and crynoics

  • Constant

    …when I realized that I share with Eliezer a strong “pro-free market ideology” I became suspicious…

    This could work both ways, though. E.g., “when I realized that I share with Eliezer the belief that creationism and intelligent design are incorrect, then I…” – then what? In some cases of sharing belief, the belief is in fact true, and should therefore raise your confidence in the other person’s judgment in other matters.

  • James D. Miller

    Constant,

    The key is whether many people whose judgement you respect also share the belief. Most economists who are smarter than me have less “faith” in free markets than either I or (I suspect) Eliezer do. This means I shouldn’t reason that Eliezer’s belief in free markets raises my estimate of his intelligence and therefore makes his singularity beliefs more likely to be true.

  • michael vassar

    James Miller: Is the concern that Eliezer has as much faith in free markets as you do? As far as I can tell from your Summit presentation, no-one has that much faith in free markets.

    Eliezer has written and spoken against unalloyed libertarianism and against specific expectations of market rationality on a number of occasions, though I can also remember personal conversations years ago where he stuck with specific wrong beliefs about financial economics after I corrected him so I think he at least formerly overestimated his econ knowledge relative to myself. He has subsequently both learned more updated though. In my estimate he still expected more from free markets than I did one year ago, but not by terribly much. He had thought less about econ and government and politics than I had and probably underestimated the failure of the econ profession to recognize that it doesn’t craft policy.

    My assessment is that an optimally regulated state would be much less market oriented than an optimally regulated achievable state as low regulation serves as an important “second-best” Schelling point. The econ profession finds the question of optimal regulation more intellectually interesting and focuses on intellectual exploration rather than in generating the best achievable policy.

  • James D. Miller

    Michael Vassar,

    Yes, compare to most everyone else my faith in free markets is extreme.

    I might be wrong, but from reading his many posts I got the impression that Eliezer is far more pro-free market than most Americans. By this I don’t mean that he thinks that markets are super-rational but rather that he has a lower estimate of the government’s ability to improve the performance of markets than most Americans do.

    ” The econ profession finds the question of optimal regulation more intellectually interesting and focuses on intellectual exploration rather than in generating the best achievable policy.” This is true as far as academic papers are concerned, although academic economists do write a lot for the popular press in somewhat practical attempts to improve regulation.

  • http://yudkowsky.net/ Eliezer Yudkowsky

    By this I don’t mean that he thinks that markets are super-rational but rather that he has a lower estimate of the government’s ability to improve the performance of markets than most Americans do.

    That pretty much nails it. My belief in the efficiency of markets has greatly decreased in the last two years; my belief in the ability of government to regulate effectively has also sunk, not risen. If I wanted to do something to improve the financial system I would start a hedge fund.

  • Constant

    …faith in free markets…a lower estimate of the government’s ability to improve the performance…

    Given the way you describe it, may I suggest replacing “I have more faith in free markets than others” with “I have less faith in government than others”. The latter more closely matches the description you give.

    Most economists who are smarter than me have less “faith” in free markets than…

    Most [a profession one of whose main paid functions and sources of prestige is to advise the state] have more faith in the state than [someone else].

    Not surprising. But few economists directly study the state, so I’m not sure that anyone should take them as authorities on the state. Public choice theory studies the state, making Buchanan and Tullock key authorities to defer to on an assessment of the state.

  • mjgeddes

    Check out ‘Betfair’ (World’s biggest betting exchange).
    Betfair

    It’s laughable how blatantly inaccurate ‘market odds’ are at gambling. You simply compare the true odds with the market odds at ‘Betfair’ and capitalize on the fluctuations. I’ve seen the odds of horses in big races fluctuate all over the place just hours before a race(typically there are fluctuations or +/- 30% above/below ‘true odds’!)

    You can actually see the echo chamber in action. What often happens is that one particular horse starts getting hyped in the racing press, and this distorts the odds of the rest of the field.

    Example: In last years’ spring horse racing carnival in Melbourne, a horse called ‘Maldivian’ was running in the Cox plate. Its true odds of winning were about 10-1, yet the Betfair ‘market odds’ on offer were a whooping 17-1 just two days beforehand. I unloaded my money on it, and…

    Maldivian Wins
    “The right time came at 4.05pm and Maldivian was sent out at $11 after being backed in from $17 since Tuesday and he gave Kavanagh and jockey Michael Rodd little cause for concern.”

    I enjoyed my free trip to Europe from that one 😀

    As to you Bayesian fan-boys, again, it’s another case of a big ‘echo-chamber’ here, with each man repeating Yudkowsky’s mantra that ‘Bayes is the secret to the universe’, and the echos reinforcing each mans blind faith…but… I’ve got a Ruby program. There are three main ‘logic’ classes in my program. (Deuducer, Predictor, and Mapper). The Predictor class has all the Bayesian stuff, and it’s a ‘wrapper’ class for Deducer class. But then I found that Mapper class wraps Predictor class, which tells me that the stuff in the former can’t be reduced to the stuff in the latter. Bayesian fan boys have been warned.

  • Matt

    1. How are you defining ‘true odds’?

    2. Fluctuations don’t imply inefficient markets. Actually all else being equal, the opposite is true.

  • mjgeddes

    1. How are you defining ‘true odds’?

    If I told you how to get those, everyone here would be rushing off to Betfair to bet on the horse races, and there would soon be no profit left for me 😉 (But see below)

    2. Fluctuations don’t imply inefficient markets. Actually all else being equal, the opposite is true.

    The factors influencing a horse race shouldn’t change that much close to a race. See my example where I got a big winner last year. You can get an idea of the ‘true odds’ by comparing the odds at ‘market close’ with the odds say two hours beforehand. The horse (Maldivian) was paying still $17 only some hours beforehand, I took a huge fixed odds contract at this price, then the price of the horse had suddenly plunged to $10 by race time! No new information should shift the odds that much, so the fluctuation must have been due to punter ignorance.

  • Matt

    “You can get an idea of the ‘true odds’ by comparing the odds at ‘market close’ with the odds say two hours beforehand.”

    Market odds at any point in time are a good estimate of true odds. The fact that the closing line is more accurate than earlier in the day is yet another point in favor of the market’s efficiency. But since we can’t travel backwards in time, it doesn’t help much with our betting.

    “No new information should shift the odds that much, so the fluctuation must have been due to punter ignorance.”

    Weather, injuries, fixes, etc. are all possibilities.

    I’m not saying there aren’t edges to be found – there obviously are. Nobody’s arguing that markets are perfect. But they do a very good job of aggregating information and making accurate predictions.

  • http://profile.typepad.com/robinhanson Robin Hanson

    This conversation has moved a bit too far away from the topic of the blog post.

  • http://profile.typepad.com/halfinney Hal Finney

    So what’s wrong with this reasoning:

    You think you’re probably right (otherwise you would change your mind). Therefore people with opinions similar to yours are more likely right than people with very different opinions. People’s accuracy on various areas tends to be correlated; being right about one thing makes it more likely that they will be right about other things. Hence people with opinions near yours tend to be more accurate in general than the average person. But opinions from people who are more accurate should be respected. Hence hearing that people who are more accurate than average also have opinions near yours should give you increased confidence in your own accuracy.

    Seems like each step is OK individually. Maybe it doesn’t quite work overall, but I could imagine something like this going on in people’s heads.

  • Jef Allbright

    @Hal Finney:

    So what’s wrong with this reasoning:
    You think you’re probably right (otherwise you would change your mind).

    Okay…

    Therefore people with opinions similar to yours are more likely right than people with very different opinions.

    Statement 2 should start with “Therefore YOU THINK people with…

  • http://profile.typepad.com/robinhanson Robin Hanson

    Hal, the approach very clearly doesn’t work in this context. If it seems to, that suggests an error of reasoning.

  • http://bustingseams.blogspot.com Breton

    There’s a key bit of missing information in your original extract of the study. You mention the heading of where the opinions came from, as presented to the subjects of the study. What isn’t mentioned is where the opinions actually came from, and how similar or dissimilar the opinions actually were to the subject’s opinions. We are left wondering whether the outcome of the study was due to the headings, or due to the opinions.

  • Pingback: Excelente blog…ti hace pnsr, creo? creo crei creere!! « Diego Gonzalez

  • Pingback: Favorecer el disenso « Ricardo Galli, de software libre