Overcoming Bias

The Comedy of Behaviorism

Followup to:  Humans in Funny Suits

"Let me see if I understand your thesis.  You think we shouldn't anthropomorphize people?"
        -- Sidney Morgenbesser to B. F. Skinner

Behaviorism was the doctrine that it was unscientific for a psychologist to ascribe emotions, beliefs, thoughts, to a human being.  After all, you can't directly observe anger or an intention to hit someone.  You can only observe the punch.  You may hear someone say "I'm angry!" but that's hearing a verbal behavior, not seeing anger.  Thoughts are not observable, therefore they are unscientific, therefore they do not exist.  Oh, you think you're thinking, but that's just a delusion - or it would be, if there were such things as delusions.

Variance-Induced Test Bias

Discussion of the Science article on gender differences in math test variance got me thinking.  Since a test score is a noisy measure of some underlying ability, an unusually high score can come either from an unusual high ability, or from an unusually positive measurement error (or both).  If higher male score variance is due more to a higher male ability variance than to a higher male measurement error variance, then a high female score is more likely to be due to measurement error than is the same high male score.  If so, treating the same score value as the same ability, independent of gender, as is common in school admissions, creates a bias (vs. men) in favor of high scoring, and against low scoring, women.

More precisely, assume that each test score s is a sum s = a + e of an ability a and a measurement error e, and that ability and measurement error are normally and independently distributed with variances A and E.  This implies test score variance is S = A + E, and that mean (and median) ability estimates given scores s are E[a|s] = m+(s-m)*(1-E/S), where m is the mean score.  The discounting factor, 1-E/S, is between 0 and 1.

Now assume men and women have the same mean score m and measurement error variance E, let R be the ratio of male to female score variance, and let N be the ratio of measurement error variance E to female score variance.  In this case, the ratio of female to male discounting factors is (1-N)/(1-N/R), which is < 1 for R > 1.   For example, if R = 1.16, the mid-estimate from the Science article, then for error fraction N values of 0.1, 0.2, or 0.4, the discounting factor ratios are 0.985, 0.967, or 0.916 -- female scores must be discounted by these factors (relative to mean scores) to be fairly comparable to male scores.  For example, applied to the math SAT (female mean 504) you'd want to subtract off (again for my sample N values) 3.7, 8.2, or 20.7 points from a 750 point female SAT score to make it comparable to a male score.  (For a 600 point score, you'd subtract 1.5, 3.2, or 8.1 points). 

No doubt there are many other factors to consider in comparing male and female candidates, but do any schools make such corrections?  Are they even aware of this bias?  Are they aware but uninterested in correcting for it?

Added 4Aug: College Admission Futures would solve this problem and many more.

Biases of Elite Education

From a thoughtful essay by William Deresiewicz:

An elite education ... makes you incapable of talking to people who aren't like you. Elite schools pride themselves on their diversity, but that diversity is almost entirely a matter of ethnicity and race. With respect to class, these schools are largely - indeed increasingly - homogeneous.  ... My education taught me to believe that people who didn't go to an Ivy League or equivalent school weren't worth talking to, regardless of their class. ... Elite universities ... select for and develop one form of intelligence: the analytic. ... social intelligence and emotional intelligence and creative ability, to name just three other forms, are not distributed preferentially among the educational elite. ... There are due dates and attendance requirements at places like Yale, but no one takes them very seriously. ... Students ... get an endless string of second chances. Not so at places like Cleveland State.  ..

Idea Futures In Lumpaland

Continuing the parable ...

Dragon markets were an exciting new fad in Lumpaland - any Oopma could bet valued cocoa beans on dragon locations, lifespans, and victim counts, and on how these might change with anti-dragon policies.  At first, instituters and amateurs both tended to dismiss markets that disagreed with favored estimates.  But as dragon markets accumulated an impressive track record, Oopmas slowly became more reluctant to disagree.  Dragon policy became better informed, more dragons were dispatched, and eventually some funds were even diverted from the institute to subsidize dragon markets. 

Yet while some instituters and amateurs came to specialize in dragon market trading, most continued with previous activities.  Most amateurs found it too tedious to specialize on particular dragons to the degree required to win bets - witty widely-read amateurs were invited to more parties.  And most instituters found that winning in the markets just did not impress to the same degree - most audiences preferred eloquence, gadgets, and math as signals of personal impressiveness. 

Prediction markets have great potential to improve info in science and business, but they'll only go as far as demand for such info.  To the extent we really care more about affiliating with impressive folks, we'll prefer activities and institutions that better achieve this end. 

Do Scientists Already Know This Stuff?

Followup to:  Science Isn't Strict Enough

poke alleges:

"Being able to create relevant hypotheses is an important skill and one a scientist spends a great deal of his or her time developing. It may not be part of the traditional description of science but that doesn't mean it's not included in the actual social institution of science that produces actual real science here in the real world; it's your description and not science that is faulty."

I know I've been calling my younger self "stupid" but that is a figure of speech; "unskillfully wielding high intelligence" would be more precise.  Eliezer₁₈ was not in the habit of making obvious mistakes - it's just that his "obvious" wasn't my "obvious".

No, I did not go through the traditional apprenticeship.  But when I look back, and see what Eliezer₁₈ did wrong, I see plenty of modern scientists making the same mistakes.  I cannot detect any sign that they were better warned than myself.

Sir Roger Penrose - a world-class physicist - still thinks that consciousness is caused by quantum gravity.  I expect that no one ever warned him against mysterious answers to mysterious questions - only told him his hypotheses needed to be falsifiable and have empirical consequences.  Just like Eliezer₁₈.

Lumpaland Parable

Once upon a time ...

Before Willy Wonka came to Lumpaland, hundreds of fierce dragons preyed on ten thousand Oompa Loompas and other wildlife.  A thousand Oopmas tried in their free time to oppose dragons, and a hundred had full time jobs at the Dragon Institute. 

Instituters were impressive - they had charisma, spoke eloquently, made cool devices and mastered hard math.  Others wanted to read about, sleep with, and study under them.  At any one time only a few instituters were out near dragons, usually at a safe distance, focused on a few relatively-safe dragons.  Some complained instituters were too distracted playing institute politics and trying to seem impressive.  But when Oopmas had to choose between an instituter and an amateur, the instituter usually won.

Amateurs were mostly content to read and argue.  And their readings and conversations rarely lingered long on one dragon.  While instituters focused on particular dragons, amateurs prided themselves on having passionate witty opinions on many dragons.  Amateurs were eager to associate with instituters, even as they complained instituters unfairly neglected their writings and favorite dragons.

The few amateurs who focused on particular dragons were considered boring, and amateurs who actually fought dragons were considered dangerous, to be avoided.  If an amateur actually managed to dispatch a dragon, the Oompas nearby trusted to report on the incident were usually instituters, who would if possible take full credit.  (If the amateur's role could not be denied, he'd be thanked for his lucky assistance to the institute's grand plan. To get more recognition, he'd have to dispatch several dragons or join the institute.)

Question: Biases afflict both amateur and instituter dragon judgments. Perhaps do-nothing amateurs are less biased, but so what?  Isn't it do-somethings' opinions that matter?

Added 16May: This story was inspired seeing amateurs indignant that professionals do not engage their brief writings on difficult complex topics (e.g., many worlds, zombies, Fermi question, nano/robot econ, disagreement, market manipulation).

Science Doesn't Trust Your Rationality

Followup to:  The Dilemma: Science or Bayes?

Scott Aaronson suggests that Many-Worlds and libertarianism are similar in that they are both cases of bullet-swallowing, rather than bullet-dodging:

Libertarianism and MWI are both are grand philosophical theories that start from premises that almost all educated people accept (quantum mechanics in the one case, Econ 101 in the other), and claim to reach conclusions that most educated people reject, or are at least puzzled by (the existence of parallel universes / the desirability of eliminating fire departments).

Now there's an analogy that would never have occurred to me.

I've previously argued that Science rejects Many-Worlds but Bayes accepts it.  (Here, "Science" is capitalized because we are talking about the idealized form of Science, not just the actual social process of science.)

It furthermore seems to me that there is a deep analogy between (small-'l') libertarianism and Science:

1. Both are based on a pragmatic distrust of reasonable-sounding arguments.
2. Both try to build systems that are more trustworthy than the people in them.
3. Both accept that people are flawed, and try to harness their flaws to power the system.

The Failures of Eld Science

Followup to:  Initiation Ceremony, If Many-Worlds Had Come First

This time there were no robes, no hoods, no masks.  Students were expected to become friends, and allies.  And everyone knew why you were in the classroom.  It would have been pointless to pretend you weren't in the Conspiracy.

Their sensei was Jeffreyssai, who might have been the best of his era, in his era.  His students were either the most promising learners, or those whom the beisutsukai saw political advantage in molding.

Brennan fell into the latter category, and knew it.  Nor had he hesitated to use his Mistress's name to open doors.  You used every avenue available to you, in seeking knowledge; that was respected here.

"- for over thirty years," Jeffreyssai said.  "Not one of them saw it; not Einstein, not Schrödinger, not even von Neumann."  He turned away from his sketcher, and toward the classroom.  "I pose to you to the question:  How did they fail?"

The students exchanged quick glances, a calculus of mutual risk between the wary and the merely baffled.  Jeffreyssai was known to play games.

Many Worlds, One Best Guess

This post is part of the Quantum Physics Sequence.
Previously in series: Collapse Postulates
Followup to:  Bell's Theorem, Spooky Action at a Distance, Quantum Non-Realism, Decoherence is Simple, Falsifiable and Testable

If you look at many microscopic physical phenomena - a photon, an electron, a hydrogen atom, a laser - and a million other known experimental setups - it is possible to come up with simple laws that seem to govern all small things (so long as you don't ask about gravity).  These laws govern the evolution of a highly abstract and mathematical object that I've been calling the "amplitude distribution", but which is more widely referred to as the "wavefunction".

Now there are gruesome questions about the proper generalization that covers all these tiny cases.  Call an object 'grue' if it appears green before January 1, 2020 and appears blue thereafter.  If all emeralds examined so far have appeared green, is the proper generalization, "Emeralds are green" or "Emeralds are grue"?

The answer is that the proper generalization is "Emeralds are green".  I'm not going to go into the arguments at the moment.  It is not the subject of this post, and the obvious answer in this case happens to be correct.  The true Way is not stupid: however clever you may be with your logic, it should finally arrive at the right answer rather than a wrong one.

In a similar sense, the simplest generalizations that would cover observed microscopic phenomena alone, take the form of "All electrons have spin 1/2" and not "All electrons have spin 1/2 before January 1, 2020" or "All electrons have spin 1/2 unless they are part of an entangled system that weighs more than 1 gram."

Caplan Pulls Along Ropes

Last May I wrote:

The space of all policies ... is huge - with thousands or millions of dimensions. ... The policy world can thought of as consisting of a few Tug-O-War "ropes" set up in this high dimensional policy space.  If you want to find a comfortable place in this world, where the people around you are reassured that you are "one of them," you need to continually and clearly telegraph your loyalty by treating each policy issue as another opportunity to find more supporting arguments for your side of the key dimensions.  That is, pick a rope and pull on it.   If, however, you actually want to improve policy ... then prefer to pull policy ropes sideways.

Bryan Caplan prefers to pull along the ropes: