Overcoming Bias

Science as Attire

Prerequisites:  Fake Explanations, Belief As Attire

The preview for the X-Men movie has a voice-over saying:  "In every human being... there is the genetic code... for mutation."  Apparently you can acquire all sorts of neat abilities by mutation.  The mutant Storm, for example, has the ability to throw lightning bolts. 

I beg you, dear reader, to consider the biological machinery necessary to generate electricity; the biological adaptations necessary to avoid being harmed by electricity; and the cognitive circuitry required for finely tuned control of lightning bolts.  If we actually observed any organism acquiring these abilities in one generation, as the result of mutation, it would outright falsify the neo-Darwinian model of natural selection.  It would be worse than finding rabbit fossils in the pre-Cambrian.  If evolutionary theory could actually stretch to cover Storm, it would be able to explain anything, and we all know what that would imply.

The X-Men comics use terms like "evolution", "mutation", and "genetic code", purely to place themselves in what they conceive to be the literary genre of science.  The part that scares me is wondering how many people, especially in the media, understand science only as a literary genre.

Media Risk Bias Feedback

Recently a friend mentioned that he was concerned about health effects from wifi. I pointed out that this was likely an overblown concern, fed by the media echoes of a scare mongering BBC Panorama program, and pointed him at the coverage at Ben Goldacre’s blog Bad Science for a through takedown of the whole issue.

To my surprise he came back more worried than ever. He had watched the program on the Bad Science page, but not looked very much at the damning criticism surrounding it. After all, a warning is much more salient than a critique. My friend is highly intelligent and careful about his biases, yet fell for this one.

There exists a feedback loop in cases like this. The public is concerned about a possible health threat (electromagnetic emissions, aspartame, GMOs) and demand that the potential threat is evaluated. Funding appears and researchers evaluate the threat. Their findings are reported back through media to the public, who update their risk estimates.

In an ideal world the end result is that everybody get better estimates. But this process very easily introduces bias: the initial concern will determine where the money goes, so issues the public is concerned about will get more funding regardless of where the real risks are. The media reporting will also introduce bias since the media favour reporting newsworthy news, and risk tends to cause greater interest than reports of no risk (or the arrival of reviews of the state of the knowledge). Hence studies warning of a risk will be overreported compared to risks downplaying it, and this will lead to a biased impression of the total risk. Finally, the public will have an availability bias that makes them take note of reported risks more than reported non-risks. And this leads to further concerns and demands for investigation.

Note that I leave out publication bias and funding bias here.There may also be a feedback from the public to media making media report things they estimate the public would want to hear about. These factors of course muddy things further in real life but mostly seem to support the feedback, not counter it.

Strangeness Heuristic

Yesterday's New York Times article on if we live in a computer simulation draws heavily from our Nick Bostrom, and at one point mentions me:

Maybe, as suggested by Robin Hanson, an economist at George Mason University, you should try to be as interesting as possible, on the theory that the designer is more likely to keep you around for the next simulation.

Interestingly, many blog reactions seem to be mainly disappointed that God might be a nerd - I guess they were hoping for a jock God.  Also interesting, blog posters seem less skeptical than blog commentors (such as the 300+ at the related NYT blog).   Apparently, blog posters defer more to the authority of the NYT, while commentors rely more on a strangeness heuristic:

Make a vivid mental picture of your best guess of how the world is, and compare that to a similar picture of someone else's claim of how the world is, was, or will be.  The larger the difference in impressions these pictures make on your mind, the less likely is the claim.

This heuristic, for example, penalizes scenarios where planes flap their wings, or where sidewalks are colored purple, or where many people walk down the street talking to small boxes.   This heuristic is relatively easy to apply and is valid on average.  So it offers a nice reference point to measure the other authorities you listen to:  For each authority, such as the NYT, the journal Nature, this blog, your own math analysis, etc., ask what is the strangest scenario that authority could convince you?

I suspect many authorities are reluctant to endorse even strongly supported strange claims, for fear of losing credibility with strangeness-heuristic-following audiences.  So bravo to the NYT here. 

What is "Public" Info?

Today's New York Times talks about cases where Wikipedia showed a death before the police or newspapers knew.   I am quoted:

Robin Hanson ... said ... these examples are hardly evidence of predicting the future. Rather, he suggested, it was "a bit newspaper-centric to say that news has not broke 'publicly' if it is being discussed online in rumors but has not appeared in a newspaper." He added that "with more and more kinds of media, there are more and more intermediate levels of info availability."

Unfortunately our laws are often based on simple-minded concepts of "public" versus "private" info.  For example, you lose your patent rights if your idea appeared anywhere on the web in any form for a year, but you can keep patent rights even if lots of other people said they already thought of your obvious idea.  Insider trading laws punished a man who traded on the "private" info of seeing a factory fire when looking out an airplane window, but as Malcom Gladwell explained, allowed a few hedge funds to escape Enron losses by carefully reading the right complex footnotes of Enron "public" reports:

Brave US TV News, Not

In the New Yorker,  Nicholas Lemann reviews James Baughman's Same Time, Same Station: Creating American Television, 1948-1961, which shows just how wrong is the view that TV news exists because citizens want an independent forth estate to courageously resist government propaganda:

The main reason [television news] came into existence, [Baughman] suggests, is that network executives, acutely aware that the United States was the only Western nation to have a predominantly private and commercial system of television, wanted to protect their berth. Broadcasters were legally required to operate in the "public interest," and they took the requirement seriously, albeit more as a meaningful threat than as a sacred duty. That's where news came in. In effect, it was a means to the end of being permitted to prosper in the entertainment business. ... Nobody should imagine that broadcasters courageously launched aggressive news divisions in the face of government hostility; if Baughman has it right, broadcasters became journalistic because the government forced them to. ...

The networks' decision to cover the quadrennial national political-party conventions, beginning in 1948, was, Baughman implies, motivated mainly by the thought that a heavily regulated industry would do well to make itself a big presence at a gathering of federal officials. The networks' commitment to documentary units, beginning in the late fifties, coincided with the aftermath of the quiz-show scandals, when it was necessary, once again, to convince public officials that there was no need to tinker with the American model of broadcasting. Even the advent of televised debates between Presidential candidates, in 1960, was, to Baughman, just "one more bone tossed to the chattering classes" by the networks' ever-fearful internal-reputation police. One can only hope that Baughman does not choose to make his next book a history of Santa Claus.

Lemann apparently does not having his illusions shattered, but I am grateful.   

Media Bias

A recent comment here by Brian expressed amusement at a citation of CBS news. Indeed, if you talk to a lay person about bias, one of the first examples that will come up is bias in the news media. A recent Zogby poll confirms widespread belief in the existence of such bias:

The vast majority of American voters believe media bias is alive and well – 83% of likely voters said the media is biased in one direction or another, while just 11% believe the media doesn’t take political sides...

I had a preconception that most people perceive bias as a function of difference from their own beliefs. Conservatives complain about liberal bias, while liberals complain about conservative bias. However the Zogby poll revealed that things are not as symmetrical as I had assumed:

While 97% of Republicans surveyed said the media are liberal, two-thirds of political independents feel the same, but fewer than one in four independents (23%) said they saw a conservative bias. Democrats, while much more likely to perceive a conservative bias than other groups, were not nearly as sure the media was against them as were the Republicans. While Republicans were unified in their perception of a left-wing media, just two-thirds of Democrats were certain the media skewed right – and 17% said the bias favored the left.

Overall, 64% perceived a liberal bias compared to 28% who see conservative bias.

Overcoming Fiction

We love fiction, in novels, movies, plays, TV, and so on.  But fiction is made up; it is not true.  Worse, storytellers have lots of standard tricks to get us to draw the conclusions they want about their fictional worlds.  Tricks use character beauty and wit, camera angles, particular consequences emphasized, and much more. 

We do try somewhat to not let fiction overly influence our beliefs about the real world.  But surely we fail in many ways; the task is just too hard.  How can we do better?

One approach is to prefer apparently true stories, such as biographies, news articles, reality TV, or grampa's war stories.   Of course lots of fiction slips in, but at least these can be fact-checked. 

Another approach is alternate versions of standard stories.  Check out The True Story of the Three Little Pigs, Wicked, and The Case for the Empire.   I'd love to see bias-reversed movies portray the same events as a famous movie, but change the standard storyteller tricks to make us draw a very different conclusion.  Imagine It's A Wonderful Life from the view of Mr. Potter, or if E.T. the Extra-Terrestrial was really a scout for an alien invasion.  The lack of interest in bias-reversed stories suggests we aren't that interested in overcoming fiction biases. 

Many people seem to think our ancestors' fiction was less realistic than ours, suggesting we are making progress toward reducing fiction biases.  I am suspicious of this claim; the fact that no one seems to have bothered to try to measure it suggests to me people don't really believe it.

Good News Only Please

Corporate prediction markets have received a lot of press over the last few years, as an exciting new trend.   But press coverage has a serious bias: it mostly only mentions success.  Hundreds of companies have considered and rejected prediction markets, and many others have started them and then stopped, at least for a while.  But news focuses on companies that are still trying them, and relatively happy. 

I used to be quoted often in those articles, until my comments became a bit more negative - that didn't fit the desired tone.  I don't blame reporters, I blame the readers; reporters understand that readers mainly want to hear breathless one-sided excitement about new technologies.  Until they don't; when over-the-top forecasts aren't verified quickly, readers may want post-fad shaking-the-head "how could they all be so gullible" articles.  After the dotcom bust, few wanted to hear about how the dotcom revolution really was continuing to remake society. 

The important lesson:  Biases that appear in the world we see via press coverage are often the biases readers expect to see, and insist on seeing.   

Outlandish Pundits

The Economist blog Free Exchange argues that journalistic pundits are biased toward outlandish predictions:

Pundits are almost never punished for being wildly wrong about something.  Nor are they rewarded for being right about something—along with 7,000 other pundits.  For journalists, a prediction pays off only if it is both right, and unusual.  This gives them an incentive to take unnecessary risks, making somewhat outlandish predictions on the off chance that they will be right.  Those pundits who "got Iraq right" or "predicted the tech bubble collapse" are feted with speaking engagements and special television appearances, while those who made sensible-but-dull arguments labour in obscurity.  The public is then surprised when their famous pundit makes some other spectacular prediction that doesn't turn out so well.

I'd add that it is not enough to just get an unusual prediction right; you have to do it with enough flair and wit and influence to be considered impressive.   I know lots of ignored people who got lots of unusual predictions right.

Press Confirms Your Health Fears

There is a huge disconnect between health factors that research suggests are most important, and health factors that get the most media and policy attention.  A new RWJF working paper suggests that the press overemphasizes obesity to satisfy reader demands:

News reports on the "obesity epidemic" have exploded in recent years, eclipsing coverage of other health issues including smoking. ... Anyone with a Body Mass Index (BMI, weight in kilos divided by height in meters squared) over 25 is deemed "overweight." ... Almost 2/3 of the U.S. population today weighs "too much" today by these standards. Recently, several researchers have argued that, for the overwhelming majority of people, weight is a poor predictor of health and should be less of a public health focus. A recent study by scientists at the Center for Disease Control and Prevention (CDC) suggests that it is only after BMI reaches 35 that there is a meaningful increase in mortality, that people in the "overweight" category actually had the lowest rate of mortality. Still, such skeptical voices remain a minority perspective in public discussion of obesity. ...

This paper exploited a unique sample of: 1) scientific articles on weight and health; 2) press releases on those studies; and 3) and news reports on those same studies ... We found that ... the news media's tendency to report more heavily on the most alarmist and individual-blaming scientific studies, and not simply how they frame individual stories, partly explains how the news dramatize and individualize science. ... These findings support the contention that scientists work as "parajournalists" writing their stories  and especially the abstract  with journalists in mind. They then frame their research via press releases and interviews with journalists. A reward structure in which, all things being equal, alarmist studies are more likely to be covered in the media may make scientists even more prone to presenting their findings in the most dramatic light possible.

The press/policy overemphasis of obesity is probably small compared to the overemphasis of medical care.  In general it is very hard for the press and academic system to tell the public anything much different from what the public expects and wants to hear.