29 Comments

http://Predictionbook.com shows how: you keep track over time, and if 60% of the time the 60% predictions occur, you are accurate. While it's statistically true that the 40% runs can be quite long, in practice that with a large enough data set you should expect the result of a 60% likely event to happen 60% of the time.

Expand full comment

Robin writes: "Can we really expect people to track the accuracy of advice from their doctors, lawyers, or interior decorators, relative to their looks, charm, and general impressiveness?"

Did you ever wonder why law firms, whose main product is catalogued and cross referenced documents, have to have 5 star locations - or geographical charm?

There are few repeat consumers of legal services - so charm and general impressiveness substitutes for reputation.

Expand full comment

Didn't Philip Tetlock write a book recently claiming that media experts have no greater accuracy than chance?

Ah, wait, it was "political" experts

http://www.amazon.com/Exper...

Expand full comment

But how would you assess the accuracy of a weather forecast? If the prediction is a 60% chance of rain tomorrow, and it turns out to be sunny, does that mean the forecast was wrong? After all, there was still a 40% chance of no rain.

Expand full comment

The study Robin references only covers television stations.

Television's primary purpose is entertainment. Content will always be secondary to presentation in this medium.

Those who care about accuracy in news and weather reporting know:

- The Internet is best. You can choose your sources, pick from a wide selection of headlines/angles, and get the latest updates. Dubious accuracy can be cross-checked.

- Newspapers come second best. The selection is space-restricted, and updates are restricted to publishing frequency. However, you can read it in your own time and skip what you don't want.

- Television is content-poor. It takes much longer to watch someone tell you about the news than to read it for yourself. You can't escape the presenter's time frame except by taping the program and replaying it later, but fast-forwarding is not like skim-reading. You either watch something or miss it. Keeping viewers engaged relies on having entertaining footage, whether this serves the content well or not.

"Why should we expect this to be any better for other kinds of news?" - The majority of people expect TV to entertain them, and doctors and lawyers to inform them. The expectation of accuracy in each case is quite different.

As for the weather; for most people, most days of the year, it really, really doesn't matter what happens. As Scott Clark said, news and weather is just for day-to-day small talk.

Expand full comment

True, most of us don't put enough thought into how accurate our news coverage is. But there are a few reasons why news outlets see it as in their interest to keep a basic level of factual accuracy:

1. Libel lawsuits.2. One news source can score an advantage over another by revealing that the other published false information (i.e. Memogate 2004 and its effects on CBS Evening News)

Of course, neither of these preclude laziness if the news source can claim good faith.

Expand full comment

Just a small addition to what Andy just said: For people who need a decent forecast, there are better sources than TV. I get my weather (when I need it) from an internet site that has proved reliable; commodities traders get theirs from people who get paid the big bucks. All the radio and TV are really good for is telling us when the feds have declared a watch or a warning; anybody can do that.

My sense of things is that the TV people err heavily on the side of predicting bad weather. This seems to align with their incentives: if they miss bad weather, the people who get caught in it will be unhappy; if they call for bad weather and it turns out OK, people will be pleasantly surprised. And their goal is to have people like them.

Expand full comment

I bet that investors in the Thai financial markets are more than happy to fly someone to Thailand to verify official reports. The television watchers are just revealing that they don't really care about the weather forecast, they just want to watch someone charming and good looking.

Expand full comment

Why such high expectations for the accuracy tracking ability or inclinations of "people"? It's not like our species makes "people" with the same quality control that we use for scientific computers or instruments.

Expand full comment

It is not a question of "accuracy". I think you're all missing the point. The news is not there to inform, but to form, to mold your opinion. And public opinion moves the world. So we are all spoon-fed lies and dishonest commentary in order to further someone's hidden agenda (see, for example, a Dan Klein paper on the intellectual dishonesty of Paul Krugman in econjournalwatch.org).

"The first and foremost of all the forces which move the world is the lie". That is the first line of Jean-François Revel's classic "La connaissance inutile" ("Useless Knowlege"), out of print everywhere (I wonder why...).

The vast majority walk around completely blind as to the forces at work in politics; a few can still discern in the penumbra weird phenomena and may pose questions about "accuracy"; but only a handful have the ability to throw light upon the darkest lies governing the world.

Expand full comment

Interesting. As a frequent bicyclist, I care a lot about the weather, but realized that in the short term, I'm better off pulling up a Doppler, than I am using anyone'e prediction of whether it is going to rain tonight. I once scoured the web looking for any studies of different weather predictions and came up completely empty. Hope I am wrong about that.

Expand full comment

"News venues that follow that innate predisposition are given more attention and, thus, make more money."

This isn't true, as the revenues of most newspapers have been declining recently, despite their scoop numbers. Also, it's interesting to question when a scoop is a scoop, since everyone knows the NY Times and the WaPo have exchanged front pages for years. Please see:

"the Post and Times send each other copies of their next day's front pages every night. The formal sharing began as a courtesy between Post Executive Editor Leonard Downie Jr. and former Times Executive Editor Joseph Lelyveld in the early 1990s and has continued ever since."

http://www.editorandpublish...

This being my third post now, I will watch further comment politely from the sidelines. Thanks for a great discussion, Robin!

Expand full comment

It's strange because I've never heard of an average viewer saying that they choose a news source because they have the most scoops, or because they report things 3 minutes earlier than the competition.

If it could cause people to say "hey, turn to channel X", if it could help the station / newspaper / whatever to develop a reputation as the source for information on sudden changes, it will be considered beneficial.

We're primed to respond to change. News venues that follow that innate predisposition are given more attention and, thus, make more money.

Of course people don't talk about things that way! People do not generally possess explicit awareness of the limitations of their cognition!

Expand full comment

frelkins, actually in this case the tv channels gave substantially different forecasts, even though they relied on the same data. But yes, editors who must choose between what "everyone knows" and what some experts tell them have incentives mostly to choose the former.

Silas, yes they don't send a new person from here for each story, but it is still very expensive to maintain foreign reporters and have them dig deep on each story.

Hal, yes, the focus on scoops is puzzling, and it is not clear what exactly the NYT has a reputation for that people like.

Expand full comment

I agree with part of what Silas says, that accuracy about Thailand is if anything even less important than accuracy about local weather. Most people's lives will not be affected at all by how accurate reporting is about Thailand (unless you live in Thailand). So there should be much less interest in accuracy on "big picture" articles like this.

One reason there may be so little attention to accuracy in local weather is that in practice there is essentially no difference in accuracy among the various reporters. Most of them are just condensing government forecasts, and it's unlikely that anyone could do noticeably better than that.

Although news organizations do not seem to compete on accuracy much, there is enormous competition on "scoops" and "exclusives". Being the first to report a story that is then picked up and widely reported elsewhere seems to be a high priority for many news organizations. It's strange because I've never heard of an average viewer saying that they choose a news source because they have the most scoops, or because they report things 3 minutes earlier than the competition.

As sa posts above, some news organizations do have a reputation for accuracy, such as the NY Times. I can't say whether it is deserved or not, but presumably this reputation is of some value to the paper and I would imagine that they take steps to guard it. People all over the country subscribe to the Times, probably in substantial part due to its reputation. Then the question is whether subscribers really care about accuracy, or whether they just like being able to say they subscribe to the Times so that they get a reputation as someone who cares.

Expand full comment

Well, Silas, I would say to some extent you are correct. Most organizations nowadays rely on a mix of local stringers, large agencies like Reuters or AP, and in areas their regional reporter (some papers still have a veteran reporter to cover the "Middle East," who probably lives in Egypt or Lebanon) or desk (if it's a big place like Paris). But if the story gains the attention of the middle or senior editors, they will in fact ask if they have the budget to send Star Reporter X over for "exclusive" coverage.

That Star Reporter X will just rely on the same stringers and a couple of taxi drivers before filing 1700 words under his own byline with an italicized "additional reporting done by. . ." at the end will pass unremarked by the section editor. Because that's just "how it's done" and you won't get far in journalism questioning conventions of this sort.

Expand full comment