Why Academics Aren’t Bayesian

5 Trackbacks

1. Social comments and analytics for this post…

   This post was mentioned on Twitter by Tony_Cookson: Reading: How academic economists are not Bayesian. http://tinyurl.com/ylg94vq...

   By uberVU - social comments November 15, 2009 at 1:12 pm

2. [...] of why things like this happen–not just in philosophy but everywhere. He proposes that academia is really about credentialling impressive minds, so ideas are treated as important if and o… For philosophy, what this means is that since it takes an impressive mind to successfully juggle [...]

   By Why decent philosophers use too much jargon : The Uncredible Hallq November 18, 2009 at 3:02 pm

3. [...] area of psychology and sociology.  It turns out that, if we are Bayesian in nature, we’re not very good Bayesians. I tend to agree with Robin Hanson on this particular subject which is to say that most people [...]

   By <corruptmemory> » blog » The futility of “direct” religious discussion November 29, 2009 at 3:36 pm

4. [...] Almost all economic models assume that human beings are Bayesians, … [but] academic economists are not Bayesians. And they’re proud of it: overcomingbias.com/2009/11/why-academics-are-not-bayesian.html [...]

   By Alexander Kruel · A Guide to Bayes’ Theorem – A few links February 27, 2010 at 6:32 am

5. [...] considering are subtle and hard to evaluate (perhaps because this policy helps us know who the impressive minds are). And in a sense the argument I’ve presented is very easy to respond to: all the theist [...]

   By Letters to Doubting Thomas (a review) : The Uncredible Hallq March 1, 2010 at 6:10 pm

25 Comments

1. Hopefully Anonymous

   November 15, 2009 at 1:45 pm | Reply

   So this is an audience distortion problem? Does that mean that many academic economists are covering, that they’re internally bayesian but externally audience pleasing?

   Does the audience distortion problem disappear when bayesian-capable economists discuss economics behind closed doors?

   Is there a race to the bottom where economists that are behaving in a non-bayesian audience optimizing fashion attack economists that deviate from audience preference by discussing economics a more bayesian way? Or is a perfectly nontransparent coordination by bayesian-capable economists?

   What does your intuition say on this?

   • Robin Hanson

     November 15, 2009 at 3:17 pm | Reply

     Well when academics have substantial incentives to actually estimate accurately, and when that conflicts with celebrating impressive folks, academics would likely prefer to do their accurate estimation in private. It is not clear that strong conflicts arise much however.

2. Andrew Gelman

   November 15, 2009 at 2:44 pm | Reply

   When youall say “Bayesian,” you’re referring to a model of rational behavior. When applied statisticians such as myself (or applied microeconomists, labor economists, etc.) say “Bayesian,” we’re talking about a particular set of statistical methods–which really have about the same claim to being rational as any other set of statistical methods do.

   To say this again: What you call Bayesian is not what I call Bayesian. And “Bayesian” is what I do for a living. This is not to say that you’re “wrong,” merely that it would make sense for you to be aware of other perspectives.

   See my bloggingheads conversation with your co-blogger Eliezer for more on this point, also I recommend you take a look at the Scholarpedia article on Bayesian statistics by David Spiegelhalter and Ken Rice.

   I use Bayesian methods myself–see my article/discussion/rejoinder in Bayesian Analysis on Objections to Bayesian Statistics for more on my motivations–but I certainly wouldn’t claim that people should be following a Bayesian approach because of any particular theoretical argument. Bayesian methods work for many problems I’ve worked on, that’s why I like them. But a method is only as good as its model. Historically there’ve been lots of Bayesians with silly models, and I think this has made some reasonable scientists wary (too wary, in fact) of Bayesian methods in general.

   • Robin Hanson

     November 15, 2009 at 3:14 pm | Reply

     I am indeed referring broadly to a model of rational belief, not more specifically a set of statistical techniques. Do you have a preferred alternative model economists should use for rational belief?

     • Andrew Gelman

       November 15, 2009 at 3:36 pm |

       I need more details to answer your question. Do you want a descriptive model of rational belief or a normative model of rational belief? If you want a model to do both, I think you’re asking for too much.

       For the descriptive model, I prefer Dave Krantz’s model of decision analysis based on goals and plans. (See also here.)

       For a normative model, I don’t really know. I would think that different people would have different normative models. For well-defined problems, I think classical utility theory has its strengths (see the chapter on decision analysis in Bayesian Data Analysis for some examples), but once you have to start patching it with things like “a taste for uncertainty,” “minimax regret,” and the like, I think it’s pretty hopeless.

       In any case, I think my above comment is relevant, because if an economist is described as Bayesian or not, this is a statement about the statistical methods that the economist uses, not a statement about their model for rational belief.

     • Robin Hanson

       November 15, 2009 at 4:19 pm |

       From the point of view of this alternative decision theory are the patterns Bryan describes, of having no opinion until something is proven or of moving estimates all the way to each new empirical estimate ignoring the previous estimates and other data, rational? If not, that seems a distraction from the main point of this post.

     • Hopefully Anonymous

       November 15, 2009 at 5:29 pm |

       Professor Gelman,
       Although I think you offer great insights, I think you’re also being a bit parasitic in your posturing relative to Prof. Hanson.

       Prof. Hanson seems to me to be in good faith reaching for ways to optimize our social epistemology, whereas it seems to me that you’re sniping at the margins here, in a way “seems a distraction from the main point of this post”

       It feels to me like your policing is in the direction of reducing cognitive ambition, that I intuit is to our detriment.

   • Steve Rayhawk

     November 15, 2009 at 4:00 pm | Reply

     One translation of “a model of rational behavior” to “a particular set of statistical methods” might be, “Every economist should update their beliefs about economics after each new study in the samy way that a rolling cumulative Bayesian meta-analysis of all studies would update its conclusions.”

     But we have no (tractable) Bayesian model of all economic phenomena, and of the conditional probabilities of studies given economic phenomena, to use in that meta-analysis. And the Bayesian model of belief updating assumes logical omniscience, so an economist would have the same beliefs before reading a mathematical proof about comparative advantage as they would afterward.

     Another better translation might be, “Every economist should update their beliefs about economics, after each new study, to the average of the beliefs that they would predict from an imagined (tractable) rolling cumulative Bayesian meta-analysis of all studies, including the new one.”

     • Robin Hanson

       November 15, 2009 at 4:07 pm |

       Don’t let the best be the enemy of the good; the standard would be to not knowingly deviate from what you expect a Bayesian would think.

     • Andrew Gelman

       November 15, 2009 at 4:32 pm |

       We have a saying in statistics: “The shitty is the enemy of the good.”

3. Eliezer Yudkowsky

   November 15, 2009 at 3:48 pm | Reply

   But surely all this is untrue of academics working in Artificial Intelligence. (Sorry, in-joke.)

4. richard silliker

   November 15, 2009 at 4:11 pm | Reply

   I would like to comment but am unable to because I lack a definition of: model of rational behavior. Can or would anybody be willing to help?

   • Hopefully Anonymous

     November 15, 2009 at 5:34 pm | Reply

     Ha!
     I attempt one on my blog. I think agent persistence optimization is the model of rational behavior of an agent.

5. Andrew Gelman

   November 15, 2009 at 6:32 pm | Reply

   I took a look at Caplan’s entry linked to above, and I can clarify that, as a practicing Bayesian statistician, I don’t think that statements of the form “Pr (hypothesis is true)” are very useful. (In Caplan’s example, he discussed “Pr (minimum wage causes unemployment).” I think you’re better off having a model of the effects of proposed policy changes (for example, an increase or a decrease in the minimum wage) and estimating the effects from there. I agree with Caplan that there can be a place for an economist’s prior distribution to enter here, but it can also make sense to try to understand what can be learned from a particular experiment. In that sense, I think things have gone the way Caplan would like. Despite the cited minimum wage study, and despite the overwhelming popularity in polls for raising the minimum wage by a lot, Congress has been reluctant to do much in that direction. Perhaps some of this reluctance can be attributed to the persuasive arguments of economic theory.

   Getting back to my earlier point, I think you should be very wary of going from the reasonable statement that people should reason logically to the highly flawed (in my opinion) statistical procedure of assigning probabilities to discrete hypotheses and trying to do Bayesian inference from there. If you want to do Bayesian inference, I think you’re better off building models, with all the costs and beneifts that has.

   And, no, in response to the other comment above, I am not trying to “posture” or “police.” As a blogger myself, I very much appreciate when people share their thoughts and comment on my blog, and I’m pretty sure that Robin feels the same way. Like “relativity” or “evolution,” he word “Bayesian” gets used in lot of ways and I think it can be helpful to understand its technical meanings.

   • Robin Hanson

     November 15, 2009 at 6:55 pm | Reply

     Seems to me Caplan is talking about an expected value: E[U|MW] – E[U|noMW]. I can’t see the point of a stat language that forbids discussing estimating such things.

     • magfrump

       November 16, 2009 at 12:27 am |

       Robin: the elimination of minimum wage would have social as well as directly economic consequences. These social consequences could give rise to different economic conditions.
       I might expect that E[U|MW] – E[U|noMW & Obama gives a speech to business owners] would be different from E[U|MW] – E[U|noMW & Obama gives a speech to the unemployed] or from E[U|MW] – E[U|noMW because they just stop enforcing it]

       In that way, having a specific policy change in mind gives us more reliable probability estimates.

       Also, I sincerely doubt that politicians’ decisions regarding minimum wages actually emerge from economic arguments (as opposed to: individual interest, political interest or sponsors, desire to appear strong/clever, pressure from party leaders). Richard, if you have some evidence of this, I would be interested to see it.

     • Andrew Gelman

       November 16, 2009 at 1:29 am |

       Robin: I have no problem with expected values or utilities (at least in theory; in practice, there can be lots of difficulties in the implementation). But if you look at Caplan’s post that you link to, there’s nothing about E and nothing about U. It’s all about P. Here’s Caplan:

       The P(minimum wage causes unemployment|Card/Krueger study’s findings). Suppose P(CK findings|m.w. does cause unemployment)=.3, P(CK findings|m.w. does not cause unemployment)=.8, P(m.w. does cause unemployment)=.99, and P(m.w. does not cause unemployment)=.01. Then the conditional probability comes out to .3*.99/(.3*.99+.8*.01)=97.4%.

       I just don’t think this particular kind of reasoning makes sense or is a helpful way to do statistics. The world (at least, the social science world) just isn’t discrete in this way.

       Anyway, you’re free to disagree with this–many statisticians do. I just wanted to isolate the way in which what Caplan (and many others) think of casually as rational Bayesiansm is not something that fits in well with my conception of Bayesian statistics.

6. Wei Dai

   November 16, 2009 at 7:29 am | Reply

   Robin, if you believe that academic belief is largely about signaling rather than truth, why do you often cite academic consensus to support your own positions, without adding a disclaimer like “but of course academic belief is mostly about signaling”? (See here and here for a couple of recent examples.) Do you expect that most of your readers are already familiar enough with signaling theory and how it applies to academia, that they can take this into account and not over-react to the evidence of academic consensus that you provide, even without an explicit disclaimer?

   • Robin Hanson

     November 16, 2009 at 8:17 am | Reply

     Most expressed human opinions are largely about signaling rather than truth. If we had a way to find what folks with expert level knowledge thought that focused mainly on a truth motivation, we’d love to defer to that. But absent prediction markets, that seems hard to find.

     • Wei Dai

       November 16, 2009 at 10:27 am |

       It does seem that academic beliefs tracks truth better than most expressed human opinions, even though they’re both about signaling.

       Markets (ordinary ones) harness rent-seeking to improve social welfare. Academia seems to harness status-seeking to improve knowledge, but perhaps in a way that we don’t understand (certainly not as well as we understand markets). Prediction markets would harness rent-seeking to improve knowledge, but interest in such institutions seem too low to make them a viable alternative to academia.

       Is anyone working on trying to better understand academia and perhaps improving it on the margins, while leaving its signaling function intact?

     • Eliezer Yudkowsky

       November 16, 2009 at 3:17 pm |

       “Rent-seeking” has a particular meaning in economics. Is there a special economic term for ordinary-desire-for-money or do we just say “greed”?

     • Hopefully Anonymous

       November 16, 2009 at 5:36 pm |

       Wei, I disagree that academia primarily harnesses status-seeking to produce knowledge. I think rather that academia is more of a coordination among knowledge seekers to optimize their knowledge-seeking. In other words, I think the credentialing of third party non-academics, etc. is to fund the knowledge-seeking, with some drift towards status driven behavior, like graduation ceremony pomp and circumstance. Faust is the academic archetype, it seems to me, like Midas is for the capitalist and Narcissus for the artist. Academia isn’t the efficient route to status. I think status-games represent a corruption of the academic, or a concession to the non-academic world.

     • anon

       November 16, 2009 at 7:08 pm |

       Eliezer, I suspect Wei Dai meant “profit-seeking”, not rent seeking. Generally speaking, rent-seeking behavior does not increase social welfare.

7. Barkley Rosser

   November 16, 2009 at 2:38 pm | Reply

   Much of this is simply due to entrenched models of teaching. So, pretty much all intro econometrics courses assume classical frequentism, and the standard BLUE theorems for such workhorse methods as OLS all assume a CF framework. It is only at a much more advanced and specialized level that one is likely to encounter Bayesian methods, particularly in time-series analysis. This is in contrast to medical research, where I understand that Bayesian methods have come close to beating out the CF ones.

   Of course, many economists when confronted with the more general arguments will grant that Bayesianism has a lot of appeal in principle. But as they only known CF models, they kind of let it go, sort of a “we are all Bayesians now, but only in theory for most of us” view.

8. michael webster

   November 17, 2009 at 9:05 am | Reply

   Robin, I thought that your signalling theory was designed to explain why rankings of prestigious institutions doesn’t correlate well with what independent rankings we have of the ability of individuals in those institutions. Am I wrong?