This paper presents the first meta-analysis for the inter-rater reliability (IRR) of journal peer reviews [using] … 70 reliability coefficients … from 48 studies. … [covering] 19,443 manuscripts; on average, each study had a sample size of 311 manuscripts (minimum: 28, maximum: 1983). … The more manuscripts that a study is based on, the smaller the reported IRR coefficients are. .. If the information of the rating system for reviewers was reported in a study, then this was associated with a smaller IRR coefficient. … An ICC of .23 indicates that only 23% of the variability in the reviewers’ rating of a manuscript could be explained by the agreement of reviewers. (more: HT Tyler)

The above is from their key figure, showing reliability estimates and confidence intervals for studies ordered by estimated reliability. The most accurate studies found the lowest reliabilities, clear evidence of a bias toward publishing studies that find high reliability. I recommend trusting only the most solid studies, which give the most pessimistic (<20%) estimates.

Seems a model would be useful here. Model the optimal number of referees per paper, given referee reliability, the value of identifying the best papers, and the relative cost of writing vs. refereeing a paper. Such a model could estimate losses from having many journals with separate referees evaluate the each article, vs. an integrated system.

Academia is primarily an institution for credentialling folks as intellectually impressive, so that others can affiliate with them.

Andrew Gelman penned “Another reason I’m glad I’m not an economist“:

That [Robin] would write such an extreme statement without even feeling the need to justify it (and, no, I don’t think it’s true, at least not in the “academia” that I know about) . . . that I see as a product of being in an economics department.

I responded:

I have posted many times here on [this]. … The standard idealistic [story] is that academics know useful and important things, things which students want to learn, media want to report, consulting clients want to apply, … These idealistic theories … have [these listed] detailed problems. … It seems far simpler to me to just postulate that people care primarily about affiliating with others who have been certified as prestigious.

Andrew answered:

College students prefer to be taught by profs who research, and hence ignore students more, yet students have little idea what their profs research. . . . There is relatively little relation between what profs teach, what profs research, and what students do after they graduate.

To which I reply: No way, dude! Our students … send me emails asking when I’m going to teach multilevel models and Bayesian statistics. … There is a strong connection between what I teach and what I research. And it’s my impression that they do use this stuff after they graduate. …

You might say: Fine, but Robin is talking about academia in general, not the Columbia statistics department in general. That I could buy … But . . . in his blog entry Robin appears to be skeptical of my claim that the customers who pay my salary “learn how to fit multilevel models.”

On why he is funded:

The state government of New York or the Heritage Foundation or whatever, … I assume they would like their conclusions to be research-based, to avoid negative unintended consequences and all the other things that we worry about when considering policies.

My primary focus is academia in general, and grad students are not “college students.”  I have consistently told both college and grad students that stat classes are among the most useful later in a non-academic career.   So I am happy to grant that Andrew may be an unusual exception.  Nevertheless, consider:

1. Since I’ve granted that my story is contrary to what people usually say and assume, saying “I assume” on funding just isn’t much of a contrary argument.  “My impression” on students isn’t much better.
2. Your funding patrons may like to see studies using your methods not because they predict better but because they and you are more prestigious.  Could you tell the difference?
3. I suspect most of your students never much use the methods they learn from you later.  Some no doubt do use them.
4. Employers may want to hire your students to use your methods not because those methods predict better because they are more prestigious, and people who can master them are just better overall.
5. Last week I mentioned that fancy stat forecasts are consistently beat by simple moving averages; have you done field tests to see how well your students actually do using your methods, compared with simpler methods?

I’d like to see some graphs of the raw data, along with replicated datasets from the model. The paper admirably connects the underlying problem to the statistical model; however, the Bayesian approach requires a lot of modeling assumptions, and I’d be a lot more convinced if I could (a) see some of the data and (b) see that the fitted model would produce simulations that look somewhat like the actual data. Otherwise we’re taking it all on faith.

But, why, if this is such a good idea, do people not do it? 

I don’t buy the cynical answer that people don’t want to falsify their own models. My preferred explanation might be called sociological and goes as follows: We’re often told to check model fit. But suppose we fit a model, write a paper, and check the model fit with a graph. If the fit is ok, then why bother with the graph: the model is OK, right? If the fit shows problems (which, realistically, it should, if you think hard enough about how to make your model-checking graph), then you better not include the graph in the paper, or the reviewers will reject, saying that you should fix your model. And once you’ve fit the better model, no need for the graph.

The result is: (a) a bloodless view of statistics in which only the good models appear, leaving readers in the dark about all the steps needed to get there; or, worse, (b) statisticians (and, in general, researchers) not checking the fit of their model in the first place, so that neither the original researchers nor the readers of the journal learn about the problems with the model.

One more thing . . .

You might say that there’s no reason to bother with model checking since all models are false anyway. I do believe that all models are false, but for me the purpose of model checking is not to accept or reject a model, but to reveal aspects of the data that are not captured by the fitted model. (See chapter 6 of Bayesian Data Analysis for some examples.)

I did my duty and wrote a letter which was published in the Journal of Theoretical Biology.  (I also emailed Kanazawa a copy but didn’t hear back from him.)  There things stood until yesterday when I saw in Tyler Cowen’s blog that Kanazawa had written an article in Psychology Today repeating the claim, "Americans who are rated "very attractive" have a 56 percent chance of having a daughter for their first child, compared with 48 percent for everyone else."  And, even more amazingly (to me), Kanazawa is publishing a book called "Why Beautiful People Have More Daughters."  The work has also been publicized in various places, including a positive mention by Stephen Dubner here (and a more mocking mention here).

OK, now to get to my question.  Kanazawa’s conjectures have not been demonstrated statistically.  (For example, the claim about beautiful parents having more daughters was barely statistically significant and was one of many possible comparisons that could’ve been done with those data.)  So it’s a little disturbing to see this as presented as "true, supported by documented scientific evidence."  On the other hand, their claim might be true.  It would be more scientifically appropriate for Kanazawa to present these results as "speculations which are supported by data," but maybe Psychology Today expects a different sort of writing?

I just don’t know how to think about this.  It’s clear to me how journalists, bloggers, and reviewers should react:  the should discuss this work with skepticism.  The trouble is that the papers were published in a reputable journal (J. Theor. Biology), and a journalist/blogger/reviewer who does not happen to see my critique would naturally tend to trust the result.  (I was only tipped off because I’d already read a bit in the area of sex ratios, for no other reason than that I’ve used boy and girl births as a teaching example.  This was essentially a bit of Bayesian reasoning by me, that Kanazawa conclusions didn’t match my priors, leading me to look more carefully at his reasoning.)  So I can’t really blame the editors of Psychology Today, or maybe even the editors at Perigee Books for not knowing any better.

But should I blame Kanazawa?  I don’t want to be dismissive of scientific speculation–I don’t like the idea of statistican as censor–so maybe there would be a way for him to present more of the full statistical story in his book (for example, in the beauty-and-daughters study, a graph with the proportion of girls born to people of all five beauty categories–rather than just comparing categories 1-4 to category 5–along with the beauty assessments from all three waves of the study).  It’s a tough call to decide how to present speculative findings.

I’m not saying that there are any easy answers to this; for example, see here for one of my struggles with interactions in an applied problem—in this case (estimating the effect of incentives in sample surveys), we were particularly interested in certain interactions even thought they could not be estimated precisely from data.

The first example comes from land navigation, which is the use of compass and map to get from one point to another.  One standard problem is to get from a point in a wood, or other occluded terrain, to a point on a road or the like, some distance away.  The unbiased approach is to take a bearing, i.e., determine a direction, from where one is to where one wants to go, and then follow it.  The problem is that as one follows the bearing, with each step a little random lateral error creeps in so that when one reaches the road one may not be sure whether the point one is seeking is to the right or the left.  The biased approach is to follow a bearing that is sufficiently to the left or right of the objective that when one reaches the road one can assume with a high degree of probability that the objective is to the right or left.

The second example comes from directing artillery fire to strike a target that one can observe, but that is an unknown distance away.  The unbiased approach is to estimate (guess) the distance, notify the gunners, observe the first shot, and then walk subsequent shots towards the target in increments of distance.  (Up 100. Up 100. etc.) The biased approach is “bracketing” the target.  In bracketing, the observer estimates (guesses) the distance, and then adds a large increment to the estimate to ensure that the first shot will fall beyond the target.  The observer then adjusts the fall of the sequence of subsequent shots by halving the distance between subsequent shots.  (ideally, by cycling through a sequence of over and under shots.  As n increases, X plus (0.5) to the nth power, times β sub n, where β is the unknown bias in estimating the unknown range X, will converge on X.  Experiments have shown that on average, bracketing will result in a faster convergence of the fire on the target than will walking.

So long as satellites and batteries don’t go dead, GPS and laser range finders now solve the land navigation and ranging problems in an unbiased manner.  Still, the questions that motivated this post remain: is the notion of useful bias itself useful?  That is, are there other, more pacific examples in the cognitive realm?

How does the distribution of truth compare to the distribution of opinion?  That is, consider some spectrum of possible answers, like the point difference in a game, or the sea level rise in the next century. On each such spectrum we could get a distribution of (point-estimate) opinions, and in the end a truth.  So in each such case we could ask for truth’s opinion-rank: what fraction of opinions were less than the truth?  For example, if 30% of estimates were below the truth (and 70% above), the opinion-rank of truth was 30%.

If we look at lots of cases in some topic area, we should be able to collect a distribution for truth’s opinion-rank, and so answer the interesting question: in this topic area, does the truth tend to be in the middle or the tails of the opinion distribution?  That is, if truth usually has an opinion rank between 40% and 60%, then in a sense the middle conformist people are usually right.  But if the opinion-rank of truth is usually below 10% or above 90%, then in a sense the extremists are usually right.

My response:

1.  As Robin notes, this is ultimately an empirical question which could be answered by collecting a lot of data on forecasts/estimates and true values.

2.  However, there is a simple theoretical argument that suggests that truth will be, generally, more extreme than point estimates, that the opinion-rank (as defined above) will have a distribution that is more concentrated at the extremes as compared to a uniform distribution.

The argument goes as follows:

Suppose that everybody’s Bayesian, everybody has the same prior distribution, but with different small amounts of data.  To give some notation:  suppose we will be looking at a sequence of parameters, theta_1, theta_2, theta_3, … with a common prior distribution p(theta), which represents the true distribution of this population of theta’s.  (We could further suppose a hierarchical structure, so that p(theta) has hyperparameters that are estimated from data, but this is not necessary for our discussion here.)  For simplicity, suppose p(theta) is a normal (bell-shaped) curve centered at 0 with standard deviation sigma.

Now suppose you get some data, y, on a parameter, theta, and summarize your inference by a point estimate which is your posterior mean, theta.hat = E(theta|y).  Averaging over all possible data y that you might see, this posterior mean a sampling distribution which is centered about 0 but with a standard deviation less than sigma.  This derives from an application of the basic variance-decomposition inequality:  var(theta.hat) = var(E(theta)|y) = var(theta) – E(var(theta|y)), which tells us that the theta.hat’s are less variable than the underlying thetas.  (This is a point we make in our paper, All Maps of Parameter Estimates are Misleading, and it also is discussed in some papers by Tom Louis.)

I posted a simulation of this (along with R code) here.  (It seemed too technical to go into this blog.)

Getting back to Robin’s question:  so, if everybody is Bayesian, using a prior distribution that correctly reflects the distribution of the underlying parameters being modeled, then, the point estimates will, on average, be closer to the center of the distribution as compared to the true values.  (To put it another way, the parameter estimates are shrunk toward the prior mean.)  And so the truth will look stranger than fiction–if fiction is thought of as point estimates!

3.  This point arises in many statistical examples:  one’s best guess is inherently more sober than what might possibly happen, which is one argument for considering fanciful possibilities in fiction. Taking your best point estimate at every step of the way will not give a realistic simulation of reality.  Reality occasionally includes the unexpected.

4.  We can apply this reasoning to sports scores, for example. Football games can be predicted to an accuracy of about 14 points (that is, the difference between the score differential and the point spread has an approximate normal distribution with mean 0 and standard deviation 14); see chapter 1 of Bayesian Data Analysis and some data here.  Looking at these data:

- The average difference between winner’s and loser’s score is 12 points.
- The average spread (point prediction of difference between winner and loser) is 5.3 points.
- 71% of the time, the score is more extreme (in difference between winner’s and loser’s score) than the spread.  (The favorite beats the spread in about half the games, and in another 20% or so of the games, the underdog actually wins by a larger margin than the favorite was favored.)
- The distribution of actual game outcomes (as measured by score differentials) is more extreme than the distribution of the point predictions.

For months, nearly everyone involved thought the medical center had had a huge whooping cough outbreak, with extensive ramifications. [...]

Then, about eight months later, health care workers were dumbfounded to receive an e-mail message from the hospital administration informing them that the whole thing was a false alarm.

Now, as they look back on the episode, epidemiologists and infectious disease specialists say the problem was that they placed too much faith in a quick and highly sensitive molecular test that led them astray.

While medical professionals can modestly improve their performance on inventories of cognitive bias when coached, we should not overestimate the extent to which formal instruction such as statistics or epidemiology classes will improve actual behavior in the field.

Measurable symmetry accounts for less than 1% of the variance in the attractiveness of women’s faces and less than 3% of the variance of the attractiveness of men’s faces.  … the initial studies showing big effects typically involved samples of less than 20 faces each, which is irresponsibly small for correlational studies with open-ended variables.  Once the bigger samples starting showing up, the effect basically disappeared for women and was shown to be pretty low for men.  But no one believed the later, bigger studies, even most of their own authors — pretty much everyone in my business still thinks that symmetry is a big deal in attractiveness.  So, the first lesson I learned:  Small samples are …  My solution has been to ditch the old p<.05 significance standard.

I see the same thing in health economics; once people see some data supporting a  theory that makes sense to them, they neglect larger contrary data.   

In the case of the Million Man March, supporters from the Nation of Islam claimed attendance of 1.5 to 2 million.  The Park Service suggested initially that 400,000 had participated.  The Malatesta Estimator therefore yields an estimate of 800,000.  We can calibrate this by comparing it with an estimate by Dr. Farouk El-Baz and his team at the Boston University Remote Sensing Lab.  Dr. El-Baz and his team used samples of 1 meter square pixels from a number of overhead photos to estimate the density per pixel, and then calculated an estimate for the entire area.  Their estimate was 837,000, with 20% error bounds giving a range from 1 million to 670,000.

Saunders, Frances Stonor. 2004. The Devil’s Broker: Seeking Gold, God, and Glory in Fourteenth-Century Italy. (New York: HarperCollins), p. 93.

BU Remote Sensing Lab Press Release: http://www.bu.edu/remotesensing/Research/MMM/MMMnew.html

Accessed 14 December 2006.