Overcoming Bias

Charm Beats Accuracy

Freakonomics:

A seven-month study of weather forecasting at Kansas City television stations was conducted over 220 days ... One [station manager] said, "There's not an evaluation of accuracy in hiring meteorologists. Presentation takes precedence over accuracy." And when discussing accuracy (or the lack thereof) of a seven-day forecast, another station manager stated, "All viewers care about is the next day. Accuracy is not a big deal to viewers." ...

The data show that stations are so consumed with ratings that accuracy in weather predictions takes an irrelevant back seat to snappy patter and charm. When directly asked if accuracy mattered in forecasting, every station manager and meteorologist said it did. But when asked what steps they had taken to measure and ensure accuracy, they were without answers.

No meteorologist or television station kept records of what they predicted, nor compared their predictions to actual results over a long term. No meteorologist posts their accuracy statistics on their résumé. No station managers use accuracy statistics in the hiring or evaluation of their meteorologists.  Instead, the focus is on charm, charisma, and presentation. Their words say they care about accuracy, but their actions say they do not.

Why should we expect this to be any better for other kinds of news?  If viewers can watch the same person day after day making predictions about something they care about and personally verify day after day, and still not care much about accuracy relative to looks and charm, how much can we really expect people to care about accuracy of news on unrest in Thailand, the credit crisis, or a new medical study?  Can we really expect people to track the accuracy of advice from their doctors, lawyers, or interior decorators, relative to their looks, charm, and general impressiveness?

Against News

Bryan Caplan raises a neglected but important issue:  are important issues neglected for news of the moment?  Bryan quotes Delos Wilcox from 1900:

But we must deplore and, so far as possible, overcome the evils of habitual newspaper reading. These evils are, chiefly, three: first, the waste of much time and mental energy in reading unimportant news and opinions, and premature, untrue, or imperfect accounts of important matters; second, the awakening of prejudices and the enkindling of passions through the partisan bias or commercial greed of newspaper managers; third, the loading of the mind with cheap literature and the development of an aversion for books and sustained thought.

Bryan also quotes Thomas Jefferson:

Avertisements contain the only truths to be relied on in a newspaper. ...
I do not take a single newspaper, nor read one a month, and I feel myself infinitely the happier for it. ...
The man who reads nothing at all is better educated than the man who reads nothing but newspapers.

Amazing Breakthrough Day: April 1st

So you're thinking, "April 1st... isn't that already supposed to be April Fool's Day?"

Yes - and that will provide the ideal cover for celebrating Amazing Breakthrough Day.

As I argued in "The Beauty of Settled Science", it is a major problem that media coverage of science focuses only on breaking news.  Breaking news, in science, occurs at the furthest fringes of the scientific frontier, which means that the new discovery is often:

• Controversial
• Supported by only one experiment
• Way the heck more complicated than an ordinary mortal can handle, and requiring lots of prerequisite science to understand, which is why it wasn't solved three centuries ago
• Later shown to be wrong

People never get to see the solid stuff, let alone the understandable stuff, because it isn't breaking news.

On Amazing Breakthrough Day, I propose, journalists who really care about science can report - under the protective cover of April 1st - such important but neglected science stories as:

• BOATS EXPLAINED:  Centuries-Old Problem Solved By Bathtub Nudist
• YOU SHALL NOT CROSS!  Königsberg Tourists' Hopes Dashed
• ARE YOUR LUNGS ON FIRE?  Link Between Respiration And Combustion Gains Acceptance Among Scientists

Against Polish

For our academic "knights in shining armor," do we care more that their suits shine, than that they are armor?   From a recent Science:

[Journal peer] reviewers make two common mistakes. The first mistake is to reflexively demand that more be done. Do not require experiments beyond the scope of the paper, unless the scope is too narrow. Avoid demanding that further work apply new techniques and approaches, unless the approaches and techniques used are insufficient to support the conclusions. ...The second mistake ... Do not reject a manuscript simply because its ideas are not original, if it offers the first strong evidence for an old but important idea. Do not reject a paper with a brilliant new idea simply because the evidence was not as comprehensive as could be imagined. Do not reject a paper simply because it is not of the highest significance, if it is beautifully executed and offers fresh ideas with strong evidence.

Most buildings have "load-bearing" beams and struts, and also extra "flourish" parts and "polish" on those parts, to help the building look good and protect it from the elements.  Similarly, intellectual writings contain both content and polish/flourish. 

My Guatemala Interviews

Last month I visited Universidad Francisco Marroquín in Guatemala and gave a series of talks and interviews.  The two interviews are available, in high definition video:

• An 11 minute interview with Luis Figueroa, on medicine.
  
• A 70 minute interview with Carlisle Johnson, on everything.

This second interview is by far the most far ranging interview I've ever had or likely will ever have, out of 200 media mentions.  Quite a credit to Mr. Johnson. 

Precious Silence, Lost

In a classic debate, two people stand toe to toe and argue, free form, about a clear claim, for a fixed time of an hour or more, in front of an attentive audience.  The best moments in such a debate are awkward "silences":  X makes a good point, and the audience can clearly see Y has no good response.  Y may change the subject, but X may soon say "But what about that point I just made, what do you say to that?"  If Y changes the subject again, an attentive audience can see clearly: Y has no good response to X's point.  Y's silence speaks volumes. 

Unfortunately, precious silences get lost in non-classic debate formats.  If the debate has no fixed end time, Y may say "sorry, I have to go now."  If there are only a few back and forth rounds, it may take most of those rounds just to clarify the claim, leaving too few rounds to clearly show a silence. 

Debates in academic journals suffer this problem; usually X speaks, Y responds, X responds again, and the debate is over.  Debates between bloggers are little better; if blogger Y responds to blogger X, usually X says nothing more.  Even when bloggers have roughly equal status, they have too many good excuses to ignore someone's point.  So the fact that a point was ignored offers only weak evidence about its strength.  When the bloggers have unequal status, it is almost hopeless; high status people usually ignore low status people, no matter how good or bad their points. 

This issue was brought home in my recent medicine debate at CATO Unbound.  I wrote a lead essay, three health policy experts of far higher academic status responded, and then we had a one week "informal discussion."  I thought I made a sharp clear claim, that crude policies to "Cut Medicine In Half" are feasible and better than the status quo.  The three respondents, however, chose to discuss other not-yet-feasible policies they liked more.  My first informal comment tried to focus them back on my claim, but two of the three had little energy for further discussion.  While they all named reasons for opposition, there were too few rounds to show clearly the weakness of those reasons.  And from blogger reactions it seems no one reads the informal discussion anyway. 

This is why I so lament, and am puzzled by, the rarity of the classic debate. 

Science as Attire

Prerequisites:  Fake Explanations, Belief As Attire

The preview for the X-Men movie has a voice-over saying:  "In every human being... there is the genetic code... for mutation."  Apparently you can acquire all sorts of neat abilities by mutation.  The mutant Storm, for example, has the ability to throw lightning bolts. 

I beg you, dear reader, to consider the biological machinery necessary to generate electricity; the biological adaptations necessary to avoid being harmed by electricity; and the cognitive circuitry required for finely tuned control of lightning bolts.  If we actually observed any organism acquiring these abilities in one generation, as the result of mutation, it would outright falsify the neo-Darwinian model of natural selection.  It would be worse than finding rabbit fossils in the pre-Cambrian.  If evolutionary theory could actually stretch to cover Storm, it would be able to explain anything, and we all know what that would imply.

The X-Men comics use terms like "evolution", "mutation", and "genetic code", purely to place themselves in what they conceive to be the literary genre of science.  The part that scares me is wondering how many people, especially in the media, understand science only as a literary genre.

Media Risk Bias Feedback

Recently a friend mentioned that he was concerned about health effects from wifi. I pointed out that this was likely an overblown concern, fed by the media echoes of a scare mongering BBC Panorama program, and pointed him at the coverage at Ben Goldacre’s blog Bad Science for a through takedown of the whole issue.

To my surprise he came back more worried than ever. He had watched the program on the Bad Science page, but not looked very much at the damning criticism surrounding it. After all, a warning is much more salient than a critique. My friend is highly intelligent and careful about his biases, yet fell for this one.

There exists a feedback loop in cases like this. The public is concerned about a possible health threat (electromagnetic emissions, aspartame, GMOs) and demand that the potential threat is evaluated. Funding appears and researchers evaluate the threat. Their findings are reported back through media to the public, who update their risk estimates.

In an ideal world the end result is that everybody get better estimates. But this process very easily introduces bias: the initial concern will determine where the money goes, so issues the public is concerned about will get more funding regardless of where the real risks are. The media reporting will also introduce bias since the media favour reporting newsworthy news, and risk tends to cause greater interest than reports of no risk (or the arrival of reviews of the state of the knowledge). Hence studies warning of a risk will be overreported compared to risks downplaying it, and this will lead to a biased impression of the total risk. Finally, the public will have an availability bias that makes them take note of reported risks more than reported non-risks. And this leads to further concerns and demands for investigation.

Note that I leave out publication bias and funding bias here.There may also be a feedback from the public to media making media report things they estimate the public would want to hear about. These factors of course muddy things further in real life but mostly seem to support the feedback, not counter it.

Strangeness Heuristic

Yesterday's New York Times article on if we live in a computer simulation draws heavily from our Nick Bostrom, and at one point mentions me:

Maybe, as suggested by Robin Hanson, an economist at George Mason University, you should try to be as interesting as possible, on the theory that the designer is more likely to keep you around for the next simulation.

Interestingly, many blog reactions seem to be mainly disappointed that God might be a nerd - I guess they were hoping for a jock God.  Also interesting, blog posters seem less skeptical than blog commentors (such as the 300+ at the related NYT blog).   Apparently, blog posters defer more to the authority of the NYT, while commentors rely more on a strangeness heuristic:

Make a vivid mental picture of your best guess of how the world is, and compare that to a similar picture of someone else's claim of how the world is, was, or will be.  The larger the difference in impressions these pictures make on your mind, the less likely is the claim.

This heuristic, for example, penalizes scenarios where planes flap their wings, or where sidewalks are colored purple, or where many people walk down the street talking to small boxes.   This heuristic is relatively easy to apply and is valid on average.  So it offers a nice reference point to measure the other authorities you listen to:  For each authority, such as the NYT, the journal Nature, this blog, your own math analysis, etc., ask what is the strangest scenario that authority could convince you?

I suspect many authorities are reluctant to endorse even strongly supported strange claims, for fear of losing credibility with strangeness-heuristic-following audiences.  So bravo to the NYT here. 

What is "Public" Info?

Today's New York Times talks about cases where Wikipedia showed a death before the police or newspapers knew.   I am quoted:

Robin Hanson ... said ... these examples are hardly evidence of predicting the future. Rather, he suggested, it was "a bit newspaper-centric to say that news has not broke 'publicly' if it is being discussed online in rumors but has not appeared in a newspaper." He added that "with more and more kinds of media, there are more and more intermediate levels of info availability."

Unfortunately our laws are often based on simple-minded concepts of "public" versus "private" info.  For example, you lose your patent rights if your idea appeared anywhere on the web in any form for a year, but you can keep patent rights even if lots of other people said they already thought of your obvious idea.  Insider trading laws punished a man who traded on the "private" info of seeing a factory fire when looking out an airplane window, but as Malcom Gladwell explained, allowed a few hedge funds to escape Enron losses by carefully reading the right complex footnotes of Enron "public" reports: