Charm Beats Accuracy

Freakonomics:

A seven-month study of weather forecasting at Kansas City television stations was conducted over 220 days … One [station manager] said, "There’s not an evaluation of accuracy in hiring meteorologists. Presentation takes precedence over accuracy." And when discussing accuracy (or the lack thereof) of a seven-day forecast, another station manager stated, "All viewers care about is the next day. Accuracy is not a big deal to viewers." …

The data show that stations are so consumed with ratings that accuracy in weather predictions takes an irrelevant back seat to snappy patter and charm. When directly asked if accuracy mattered in forecasting, every station manager and meteorologist said it did. But when asked what steps they had taken to measure and ensure accuracy, they were without answers.

No meteorologist or television station kept records of what they predicted, nor compared their predictions to actual results over a long term. No meteorologist posts their accuracy statistics on their résumé. No station managers use accuracy statistics in the hiring or evaluation of their meteorologists.  Instead, the focus is on charm, charisma, and presentation. Their words say they care about accuracy, but their actions say they do not.

Why should we expect this to be any better for other kinds of news?  If viewers can watch the same person day after day making predictions about something they care about and personally verify day after day, and still not care much about accuracy relative to looks and charm, how much can we really expect people to care about accuracy of news on unrest in Thailand, the credit crisis, or a new medical study?  Can we really expect people to track the accuracy of advice from their doctors, lawyers, or interior decorators, relative to their looks, charm, and general impressiveness?

GD Star Rating
loading...
Tagged as:
Trackback URL:
  • sa

    Your question is correct but misses a main point. If the viewers don;t verify acciracy then it’s not important to anyone and it is just as well that no one bothers. Unrest in Thailand matters more to some (but not all) people and presumably it get cross-checked by those in the know.

    The author of the study here measures a market failure but it’s hard to see where the market was at all before this.

  • Silas

    Why should we expect this to be any better for other kinds of news?

    Because predicting what future weather will be, is a lot harder than predicting what happened in the recent past?

  • Nick Tarleton

    Besides what Silas says, most news is gathered and presented by different people.

  • http://hanson.gmu.edu Robin Hanson

    Silas and Nick, do you have any idea how much it costs to have someone fly to Thailand and then dig around to independently check on claims made in official government reports?

    sa, the whole question is: can those who are not “in the know” find out by reading the newspaper?

  • Silas

    Robin_Hanson: It costs a lot to fly someone to Thailand[1], but you weren’t comparing “government pronouncements” to reality; you were comparing press accounts to reality.

    The question is: can people be expected to care about truth over looks for Thailand news, when they value looks over truth is weather reporting?

    Correctly reporting the situation in Thailand is easier than correctly guessing the weather. Plus, reporters know they can be caught in a lie if others report differently. A rational person could spend some weather accuracy, which isn’t high to begin with, to get a pretty face to look at, while not being willing to spend news accuracy to get a pretty face.

    Now, I actually think people are deceived by looks for doctors, lawyers, and even often reporters and analysts. I’m just saying (to put it in Bayesian terms):

    P(favor looks over accuracy in weather | favor looks over accuracy in news) is high, but so is

    P(favor looks over accuracy in weather | not favor looks over accuracy in news).

    [1] but you can get a great deal if you’re flexible on flight times.

  • michael vassar

    Robin: I think that this posts is one of your best presentations of one of your major important themes. I’ll be using it in conversation. Thanks.
    I want to mention again my thanks for the huge contribution you have made to my way of thinking about the world over the years.

  • http://hanson.gmu.edu Robin Hanson

    Silas, you’ve lost me. If our news about Thailand is to be accurate, some reporter has to fly to Thailand and dig around to find out the truth. If is far easier for that reporter to just stay home and read an official Thai goverment report on the unrest. The question is why we should expect reporters to go to all that extra work if readers don’t much care about accuracy. If readers care more about wit, why not stay home and write up a witty article relying on the government report?

  • Caledonian

    If our news about Thailand is to be accurate

    People don’t care about accurate news.

    If readers care more about wit, why not stay home and write up a witty article relying on the government report?

    Because people want to be perceived as valuing accuracy and truth, and news stations want to be perceived as accurate. If they go to entertaining but easily-exposed counterfactual fluff, both those things are put in jeopardy.

    In some cases, particularly with ideological positions, people want their ideology to be considered ‘truth’, and so care more about whether the programming is compatible with their beliefs than whether they’re even loosely plausible.

  • scott clark

    Robin is dead on accurate here, as usual,I would add. But I don’t think he is saying anything controversial.

    Warning: Massive Generalities Follow.

    People watch the weather report, not so they can know what the weather will be, but so they have something handy to say when making idle small talk. People watch the news, not so they know insightful truths about current events, but so they have something handy to say when the idle small talk gets turned up a notch. Accuracy has little to do which it, as long as it is handy in a conversation and makes you look like you are on the ball, and up to speed.

    We know style wins over substance, big personalities get their own CNN or MSNBC or FOXNews show. People also love to get their news wrapped up in comedy from the Daily Show and the Colbert Report. And the more hilarious it is, the less accurate it needs to be. The more bluster a big personality can put around a story, the less developed the story has to be, it gets to a point where it is all bluster and no story, because, for those with the talent, it is way easier to just bluster, then report nuance.

    You absoutely correct about this, but this shouldn’t come as a suprise.

  • http://www.iphonefreak.com frelkins

    An interesting thing about weather, according to a fairly recent study, is that if you just say “tomorrow will be like today,” you will be right about 75% of the time. Of course I read that study in the NY Times, so I don’t know how accurate the report of the findings were.

    Most local TV stations just buy weather reports from the government or other providers and have only a few capabilities of their own – as a result, most weather reports are going to be the same because they rely on the same data. Thus accuracy isn’t important to them, because their competitor won’t be any more or less accurate than they are/are not. That’s why charm is the distinguishing factor here, and why no one keeps statistics on accuracy. They don’t care about accuracy, only about ratings, which is how they sell ads and make money.

    As for news accuracy overall, my personal experience (I have been interviewed many times myself by large organizations) is that the bias of the editors is unseen and overwhelming. Not that my personal experience is useful here, but I will remark that when scientific information I possessed contradicted the conventional wisdom the editors “knew,” I was discredited as a source every time without backup reporting, even when I provided the evidence. Because “everyone knows” what the editors “know” and they quickly discard what is “obviously untrue.”

    As a result, I have zero trust in the NY Times.

  • sa

    @Hanson

    Not everyone cares about every type of news. But there are ways to mitigate this bias towards to under-prioritise accuracy.

    When you are reading the NYT you can be reasonably sure of it’s accuracy. When you are reading Consumer Reports you can be reasonably sure of it’s opinions.

    If readers really care so much then they will do thier homework. You are saying (and you say this a lot) that people make up their minds based on the drivel they read in the newspapers. I say that, what you say is correct AND ill-informed people should not be engaged with. You want to engage them and therefore feel frustrated(rightly so).

    In the end it all boils down to the same old thing: put your money where your mouth is and you will suddenly see a lot more honesty and a lot more silence.

  • Mason

    I think Caledonain is correct, people want to be perceived as valueing accuracy (triple doplar anyone?) and that’s why news stations fly someone out to Thailand. While the reporter is there s/he probably doesn’t spend much time looking for facts, compared to the amount of time spent looking for great footage. Should some facts come up along the way, might as well report them (assuming they don’t clash with the visual).

  • Silas

    Robin_Hanson: You seem to have vast misconceptions about how news reporting works. The image of “Hey, something is happening in Thailand, do we have budget to send someone over?” is unrealistic. Various news organizations have people stationed in various places across the world, and in any given country, you’ll find at least one or two reporters for various news agencies. If something comes up, that news agency will ask for an update. Yes, the reporter can just copy what the government says, but that risks being disproven by a more honest reporter stationed in the same country.

    Now, I still accept some of your more general point — people may not care about accuracy in news. Like your friend Bryan Caplan argues in The Myth of the Rational Voter, people will indulge irrationality until it’s expensive. My point is, there are good reasons to trade rationality for a pretty face on weather reports — you’re not giving much up! And people may trade away rationality on news too — because while reporters can get accuracy more easily (unlike predicting the weather), it doesn’t affect the end consumer’s life. My point is that the former (weather) doesn’t carry the implications you claim for the latter (news). To the extent that people act the same way, it’s for different reasons. (Low accuracy expectations in the former, low cost of acting on wrong information in the latter.)

  • http://www.iphonefreak.com frelkins

    Well, Silas, I would say to some extent you are correct. Most organizations nowadays rely on a mix of local stringers, large agencies like Reuters or AP, and in areas their regional reporter (some papers still have a veteran reporter to cover the “Middle East,” who probably lives in Egypt or Lebanon) or desk (if it’s a big place like Paris). But if the story gains the attention of the middle or senior editors, they will in fact ask if they have the budget to send Star Reporter X over for “exclusive” coverage.

    That Star Reporter X will just rely on the same stringers and a couple of taxi drivers before filing 1700 words under his own byline with an italicized “additional reporting done by. . .” at the end will pass unremarked by the section editor. Because that’s just “how it’s done” and you won’t get far in journalism questioning conventions of this sort.

  • http://profile.typekey.com/halfinney/ Hal Finney

    I agree with part of what Silas says, that accuracy about Thailand is if anything even less important than accuracy about local weather. Most people’s lives will not be affected at all by how accurate reporting is about Thailand (unless you live in Thailand). So there should be much less interest in accuracy on “big picture” articles like this.

    One reason there may be so little attention to accuracy in local weather is that in practice there is essentially no difference in accuracy among the various reporters. Most of them are just condensing government forecasts, and it’s unlikely that anyone could do noticeably better than that.

    Although news organizations do not seem to compete on accuracy much, there is enormous competition on “scoops” and “exclusives”. Being the first to report a story that is then picked up and widely reported elsewhere seems to be a high priority for many news organizations. It’s strange because I’ve never heard of an average viewer saying that they choose a news source because they have the most scoops, or because they report things 3 minutes earlier than the competition.

    As sa posts above, some news organizations do have a reputation for accuracy, such as the NY Times. I can’t say whether it is deserved or not, but presumably this reputation is of some value to the paper and I would imagine that they take steps to guard it. People all over the country subscribe to the Times, probably in substantial part due to its reputation. Then the question is whether subscribers really care about accuracy, or whether they just like being able to say they subscribe to the Times so that they get a reputation as someone who cares.

  • http://hanson.gmu.edu Robin Hanson

    frelkins, actually in this case the tv channels gave substantially different forecasts, even though they relied on the same data. But yes, editors who must choose between what “everyone knows” and what some experts tell them have incentives mostly to choose the former.

    Silas, yes they don’t send a new person from here for each story, but it is still very expensive to maintain foreign reporters and have them dig deep on each story.

    Hal, yes, the focus on scoops is puzzling, and it is not clear what exactly the NYT has a reputation for that people like.

  • Caledonian

    It’s strange because I’ve never heard of an average viewer saying that they choose a news source because they have the most scoops, or because they report things 3 minutes earlier than the competition.

    If it could cause people to say “hey, turn to channel X”, if it could help the station / newspaper / whatever to develop a reputation as the source for information on sudden changes, it will be considered beneficial.

    We’re primed to respond to change. News venues that follow that innate predisposition are given more attention and, thus, make more money.

    Of course people don’t talk about things that way! People do not generally possess explicit awareness of the limitations of their cognition!

  • http://www.iphonefreak.com frelkins

    “News venues that follow that innate predisposition are given more attention and, thus, make more money.”

    This isn’t true, as the revenues of most newspapers have been declining recently, despite their scoop numbers. Also, it’s interesting to question when a scoop is a scoop, since everyone knows the NY Times and the WaPo have exchanged front pages for years. Please see:

    “the Post and Times send each other copies of their next day’s front pages every night. The formal sharing began as a courtesy between Post Executive Editor Leonard Downie Jr. and former Times Executive Editor Joseph Lelyveld in the early 1990s and has continued ever since.”

    http://www.editorandpublisher.com/eandp/news/article_display.jsp?vnu_content_id=1001136170

    This being my third post now, I will watch further comment politely from the sidelines. Thanks for a great discussion, Robin!

  • LB

    Interesting. As a frequent bicyclist, I care a lot about the weather, but realized that in the short term, I’m better off pulling up a Doppler, than I am using anyone’e prediction of whether it is going to rain tonight. I once scoured the web looking for any studies of different weather predictions and came up completely empty. Hope I am wrong about that.

  • Fabio Franco

    It is not a question of “accuracy”. I think you’re all missing the point. The news is not there to inform, but to form, to mold your opinion. And public opinion moves the world. So we are all spoon-fed lies and dishonest commentary in order to further someone’s hidden agenda (see, for example, a Dan Klein paper on the intellectual dishonesty of Paul Krugman in econjournalwatch.org).

    “The first and foremost of all the forces which move the world is the lie”. That is the first line of Jean-François Revel’s classic “La connaissance inutile” (“Useless Knowlege”), out of print everywhere (I wonder why…).

    The vast majority walk around completely blind as to the forces at work in politics; a few can still discern in the penumbra weird phenomena and may pose questions about “accuracy”; but only a handful have the ability to throw light upon the darkest lies governing the world.

  • http://www.hopeanon.typepapd.com Hopefully Anonymous

    Why such high expectations for the accuracy tracking ability or inclinations of “people”? It’s not like our species makes “people” with the same quality control that we use for scientific computers or instruments.

  • Andy

    I bet that investors in the Thai financial markets are more than happy to fly someone to Thailand to verify official reports. The television watchers are just revealing that they don’t really care about the weather forecast, they just want to watch someone charming and good looking.

  • Alan Gunn

    Just a small addition to what Andy just said: For people who need a decent forecast, there are better sources than TV. I get my weather (when I need it) from an internet site that has proved reliable; commodities traders get theirs from people who get paid the big bucks. All the radio and TV are really good for is telling us when the feds have declared a watch or a warning; anybody can do that.

    My sense of things is that the TV people err heavily on the side of predicting bad weather. This seems to align with their incentives: if they miss bad weather, the people who get caught in it will be unhappy; if they call for bad weather and it turns out OK, people will be pleasantly surprised. And their goal is to have people like them.

  • Patrick

    True, most of us don’t put enough thought into how accurate our news coverage is. But there are a few reasons why news outlets see it as in their interest to keep a basic level of factual accuracy:

    1. Libel lawsuits.
    2. One news source can score an advantage over another by revealing that the other published false information (i.e. Memogate 2004 and its effects on CBS Evening News)

    Of course, neither of these preclude laziness if the news source can claim good faith.

  • Wendy Collings

    The study Robin references only covers television stations.

    Television’s primary purpose is entertainment. Content will always be secondary to presentation in this medium.

    Those who care about accuracy in news and weather reporting know:

    - The Internet is best. You can choose your sources, pick from a wide selection of headlines/angles, and get the latest updates. Dubious accuracy can be cross-checked.

    - Newspapers come second best. The selection is space-restricted, and updates are restricted to publishing frequency. However, you can read it in your own time and skip what you don’t want.

    - Television is content-poor. It takes much longer to watch someone tell you about the news than to read it for yourself. You can’t escape the presenter’s time frame except by taping the program and replaying it later, but fast-forwarding is not like skim-reading. You either watch something or miss it. Keeping viewers engaged relies on having entertaining footage, whether this serves the content well or not.

    “Why should we expect this to be any better for other kinds of news?” – The majority of people expect TV to entertain them, and doctors and lawyers to inform them. The expectation of accuracy in each case is quite different.

    As for the weather; for most people, most days of the year, it really, really doesn’t matter what happens. As Scott Clark said, news and weather is just for day-to-day small talk.

  • Jamie Forrest

    But how would you assess the accuracy of a weather forecast? If the prediction is a 60% chance of rain tomorrow, and it turns out to be sunny, does that mean the forecast was wrong? After all, there was still a 40% chance of no rain.

  • Newton

    Didn’t Philip Tetlock write a book recently claiming that media experts have no greater accuracy than chance?

    Ah, wait, it was “political” experts

    http://www.amazon.com/Expert-Political-Judgment-Good-Know/dp/0691128715

  • http://bizop.ca michael webster

    Robin writes: “Can we really expect people to track the accuracy of advice from their doctors, lawyers, or interior decorators, relative to their looks, charm, and general impressiveness?”

    Did you ever wonder why law firms, whose main product is catalogued and cross referenced documents, have to have 5 star locations – or geographical charm?

    There are few repeat consumers of legal services – so charm and general impressiveness substitutes for reputation.