Suspiciously Vague LHC Forecasts

Me in ’06:

You can get 80% of the improvement that prediction markets offer by using a much simpler solution: collect track records.  … When people make forecast-like-statements, write them down in a clear standardized form, and then check back later to see who was more accurate.

I’m at scifoo (Nature/O’Reilly/Google Science Foo Camp) and yesterday heard a talk about the Large Hadron Collider that will go live in a few weeks – and had a disturbing thought.  Odds are very good that within the next few years we will see news articles where bigshot physicists say a new LHC result vindicates a theory they’ve been pushing.  But today there are no public predictions by high-profile physicists stated precisely enough to be clearly scored for accuracy!  While weather, business, and sport forecasters commonly make scoreable probability forecasts, here are the sorts of forecasts bigshot physicists make:

Randall: An answer to the question of the weakness of gravity … should be revealed.
Lederman: Will surely help us to understand  … the accelerated expansion of the universe.
Vilenkin: [If] supersymmetry was responsible for the apparent fine-tunings in particle physics … then signatures of supersymmetry are very likely to show up at LHC.
Rees: I’m hoping that it will clarify the nature of the particles that constitute the "dark matter."
Witten: A process of "symmetry breaking" [between electromagnetic and weak interactions] … whereby nature spontaneously picks one force over another – even though fundamentally they are equivalent. The LHC will tell us whether this notion is correct, and if so, how it works.
Susskind: I see only two possible outcomes of the LHC project – either there will be low energy supersymmetry, or there won’t. If there isn’t, I would expect that the minimal Standard Model will prevail. In either case, the Higgs particle … will be shown to exist.
Schwarz: Indications of extra dimensions, black holes, strings, magnetic monopoles, etc.  … I am pessimistic about the prospects for finding them in the LHC’s energy range.
Kane: The LHC data could test supersymmetry, establish string theory .

Sean Carroll at least attaches probabilities:

The Higgs Boson: 95%.  … Supersymmetry: 60%. …  Large Extra Dimensions: 1%. … Warped Extra Dimensions: 10%. … Evidence for or against String Theory: 0.5%. … Dark Matter: 15%. … Dark Energy: 0.1%. … Strong Dynamics: 5%. … New Massive Gauge Bosons: 2%. … New Quarks or Leptons: 2%. … Preons: 1%. … Mysterious Missing Energy: 15%. … Baryon-Number Violation: 0.2%. … Magnetic Monopoles, Strangelets, Q-Balls, Solitons: 1%. … Unparticles: 0.5%. … Something that Has Never Been Predicted: 50%. … Something that Has Been Predicted, but Not Listed Above: 2%. … Absolutely Nothing: 3%

What the LHC will more directly find are particles with particular properties, such as charge, spin, mass, lifetime, and source and sink particle decay rates.  Translating such particle property packages to the labels above (e.g. "supersymmetry") will take some interpretation, allowing lingering and perhaps unsettleable disputes about which predictions were vindicated.  So why don’t bigshot physicists make clear scoreable forecasts, i.e., about probabilities of particle property packages?

It is not that bigshot physicists are simplifying for a popular audience – at scifoo camp I asked many of them (and some Nature editors) directly, and none knew of any clear scoreable forecasts.  It is not that the scoring would be too noisy to worth bothering – there seem to be enough different things one could forecast here to collect a reasonable score from someone’s forecast over all of them.   And while many emphasized that masses are hard to predict, we know how to score conditional forecasts – e.g., tell us about other properties conditional on a given mass range.

Admittedly, part of the problem is that the space of particle property packages has high dimension, and so volumes that theories favor may not be simply described in terms of single dimensions in this space – it may take some care to express which volumes have what probabilities.  But geez – the LHC costs over $10 billion.  Shouldn’t we expect bigshot physicists who will want to later crow that the LHC vindicated their theories to bother to devote a little time expressing their predictions in a scoreable form?  We don’t accept less from weather, business, or sport forecasters – why accept less from physicists?

Added 12Aug:  Yes particular papers (e.g. here) may offer point estimates or bounds.  But bigshots who hope to crow later should say now which (of 100s+) papers out there they endorse, and with what probabilities. 

GD Star Rating
Tagged as:
Trackback URL:
  • Rolf Andreassen

    Unfortunately, particle theory at the moment is in the situation of flinging a lot of mud at the universe and hoping some of it sticks. Charges and spins are fairly trivial to predict – the scientists you talked to probably didn’t think them worth mentioning, although they are nice and testable and do differ between models. Masses and branching fractions (and therefore lifetimes) are basically impossible to predict except in the loosest sense of “the lightest X particle will live longer than its excited states because its decay requires breaking of X symmetry; the excited states will decay by (the fastest possible interaction that doesn’t break X symmetry)”. We don’t have a good mathematics for describing what’s going on at the quark level; we’re reduced to doing a bunch of empirical descriptions of what we see, and extrapolating to higher energies. So much of what the LHC will tell us is which simplifications and approximations were reasonable to make, in the sense of still providing a good description at higher energies. Once we know that we can take another stab at providing an underlying model.

    Another reason for the lack of predictions is that most papers on this take the form “assume X; then we will see Y dependence of Z on theta.” These aren’t predictions that X is true, they are proposed ways to falsify X – if you don’t see Y, then X is untrue.

  • mike kenny

    do you think it would be possible or useful for others to attach probabilities to expert predictions? that is, if the experts won’t give probability estimates, we take public comments and formulating them in terms of probability estimates ourselves? this might force experts to say ‘don’t put words in my mouth,’ to which one could say ‘well, am i estimating too high or too low?’

    maybe such interpretations of probability could be posted on the web. i’m just thinking there must be some comments that are made, perhaps less publicly, by experts that one could reasonably attach probability estimates to. ‘i’d be surprised if x happened’ would = 90% likelihood of x not happening, say.

    i suppose the estimates could be based on past remarks of experts too, like ‘when dr. jon doe says something like “i’d be surprised if x happens,” it historically seems to means there’s about a 90% probability of it not happening, but when dr. jane doe says the same thing, it means there’s a 95% chance that x won’t happen.’

    so experts could recalibrate their language if their probability estimates are different from their record, but also listeners could recalibrate their interpretation of expert estimates, even if experts didn’t recalibrate their language.

  • Tim Tyler

    Scientific theories and hypotheses normally represent the predictions of scientists. They do not necessarily look much like predictions – but that’s just how scientists usually do things.

  • Nancy Lebovitz\

    Maybe prediction is hard in physics precisely because it’s real exploration. And physicists are smart enough to know it.

  • mike kenny

    i’m thinking maybe a clever group of programmers/social scientists/statisticians/linguists could maybe put together some kind of program that could examine a sample of human language to determine the implicit probability estimates of certain claims. ‘on average when a person says ‘x is impossible’ the impossible thing happens y% of the time,’ et c. the internet provides a lot of claims to be scoured! wouldn’t it be great if you could draw on that info when someone is telling you something is ‘impossible’ or ‘highly likely’ et c?

  • Tim Tyler

    Re: Sean Carroll at least attaches probabilities: The Higgs Boson: 95%. … Supersymmetry: 60% … Something that Has Never Been Predicted: 50%. … Absolutely Nothing: 3%

    3%? I don’t think Sean Carroll’s probabilities add up.

  • Aspiring Vulcan

    The probabilities should add up only if they are mutually exclusive events.

  • Tim Tyler

    You are taking “add up” too literally. I drew attention to the 3% for a reason.

  • steven

    Mostly I agree, but I worry that we might confuse the predictions made by a physics theory with the predictions made by the theory’s advocates.

  • Douglas Knight

    Robin Hanson:
    allowing lingering and perhaps unsettleable disputes about which predictions were vindicated

    Has this ever happened in physics?
    say, 2 years of dispute over which theory was vindicated?
    Susy & Higgs do not have much danger of disputed interpretation, as I see it.

    I think Sean Carroll has the outlines right: there are two predictions (susy & Higgs) which have a reasonable chance of being confirmed and the rest is a lottery. Those are poor incentives for public predictions.

    Incidentally, his predictions are a textbook case of bias towards the specific and against the general.

  • JTankers

    The estimate above for evidence of string theory at 0.5%, I assume that was code for micro black hole creation.

    Abstract below from Dr. Rossler’s plea to the world, copy available on

    Arrogant of the scientist to rely on dubious at best safety arguments and pretend not to hear inconvenient warnings and more credible arguments.

    “A nightmarish situation, that can still be hoped to be averted in time through communication within the scientific community, is drawn attention to. Only a few weeks remain to find out whether the danger is real or nothing but a mirage. After this time window is closed, it will take years until we know whether or not we are doomed. The story line has all the features of a best-selling novel. The reader is asked to contribute constructively.”

    Quote from Dr. Otto E. Rossler, Professor Theoretical Biochemist, visiting Professor of Theoretical Physics, inventor of the Rossler Attractor, founder of Endophysics, winner of the 2003 Chaos Award of the University of Liege and the 2003 Rene Descartes Award.

    Full text at:

  • thereisnospoon

    so basically the ‘scientific community’ doesn’t have a clue.

    they need the lhc to explain the same theory they use to tell to the public ‘don’t worry. we know what we’re doing. see, according to this book if X then Y and Z. but we need to fire this thing up to confirm the book first’. great job.

    if there is a big filter at least we’ll be in the hall of fame of the most moronic civilizations to ever pass through it. i salute you.

  • Ben Jones

    Someone tell me where I can put £20 on Q-Balls. Just got a feeling.

  • Robin Hanson

    Rolf, so can you offer some scoreable predictions about charges and spins?

    Mike, David Brin has been making proposals like that. I’m skeptical.

    steven, we almost never see theories by themselves – the forecasts we get when theories are actually applied usually include physicists judgment about auxiliary factors. So I don’t see the problem with doing the same here to get forecasts from the theories.

  • Rolf Andreassen

    > can you offer some scoreable predictions about charges and spins?

    Well, I’m not a theorist, but supersymmetry offers us sleptons, which have the same charge as leptons (electron, muon, tau) but no spin. There are likewise supersymmetric partners for each of the other SM particles. This is not an exotic theory by any means, you can look up the spin-charge predictions on Wikipedia. This is pretty easy to score, or at least it would be if you knew that the various detectors of the LHC were sensitive to individual particle spins, which is not obvious. With an average event multiplicity of several thousand, it’s not easy to pick out individual properties.

  • Sean Carroll

    I don’t really think the comparison with sports/business/weather forecasters really holds up, for a prosaic reason — in particle physics, the timescale for experiments is years and decades, not days. There is no way to efficiently grade/reward people on the accuracy of their predictions, and correspondingly no real incentive for anyone to make very quantitative predictions.

    On the other hand, it’s not as if there is no incentive to be right. If you devote your life to working out the ramifications of low-energy supersymmetry and it’s not there, you won’t get fired (if you have tenure), but on the other hand your life’s work will be useless. Which is a pretty big incentive.

  • Robin Hanson

    Sean, I don’t understand the relevance of the timescale to the efficient grading of predictions. Given enough forecasts we can see a signal of accuracy above the noise of luck in individual forecasts. I agree that the longer the timescale the weaker are incentives from any given reward tied to scoring. But I’m not really focused on incentives in this post – I’m focused on whether it is reasonable for folks to crow about being vindicated when they weren’t willing to make scoreable forecasts.

  • onymous

    I don’t understand what you’re asking for. There are zillions of papers that explain what the LHC would see given a particular model. Those are making predictions. But of course for most of those models no one expects them to be exactly right, not even the authors. I don’t see how to attach any meaningful numerical “confidence” to any prediction.

  • wait a minute

    There are mass estimates of the Higgs boson by non-string theorists with probability bars.

    Try digging a little deeper.

  • onerock

    sean carroll, who’s funding the lhc? $10bln for a hunch seems pretty unlikely to me when they shoot down hubble because it’s expensive. there must be something else there.

  • mike kenny

    thanks for the tip robin. i’ll check out david brin. are there good places to look for criticisms of such approaches off the top of your head?

  • Midas Oracle .ORG

    FAMOUS SCIENTIST TO ROBIN HIGH IQ HANSON: Science, which is a very long-term endeavor, does not need your stickin idea about scoreable predictions and track records. Please, go back to minding economic issues in your Ivory Tower, and let us run science ou

    Overcoming Whatever:
    I dont really think the comparison with sports/business/weather forecasters really holds up, for a prosaic reason in particle physics, the timescale for experiments is years and decades, not days. There is no way to …

  • Robin Hanson

    onymous, if people want to crow about being vindicated, then people must make forecasts. It is not enough that there were 1000 papers that each made a forecast, and then crowing in retrospect about the one paper that got closest – bigshot physicists should choose now which of those paper forecasts to endorse to what degree.

  • Rolf Andreassen

    Well, I think this is sort of the point – physics doesn’t have any information to choose one model over another. If we did, we wouldn’t have to do the experiment. It’s all of the form “if X, then Y”, with nothing to tell you how to choose between the X other than seeing which Y you get. Nobody’s going to stake any reputation on a gut feel.

  • Eliezer Yudkowsky

    If you can’t say which theories predict which experimental results now, how dare you say so after peeking at the LHC results?

  • Rolf Andreassen

    I don’t think that’s what I said. “If X, then Y” is a prediction. But we have no means of favouring that prediction over “If A, then B”. If we did, we wouldn’t need the experiment. So, nobody is going to come out and say “A is true, so we will see B”. There’s a difference between believing “If A then B” and believing that A is more likely than X.

  • Garrett

    Hi Robin, it was good to meet you at SciFoo. To generalize your post, I think it would be very cool to have a technical futures market, where anyone could bet money on scientifically testable predictions. But, barring that, Sean’s probabilities seem about right, though mine differ a little. In particular, my horse in this race is consistent with the appearance of a Higgs, a W’ and Z’, and some colored bosons. If some of these are seen (which Sean guestimates at less than a 2% chance) I will be extremely happy, and probably crow a bit. If anything else appears at the LHC, my horse and crow are probably cooked.

  • Robin Hanson

    Garrett, was great to meet you and to realize you actually knew who I was! 🙂 Good to hear you make predictions, but yours aren’t quite scoreable yet, as you neglected to assign them probabilities.

  • Hopefully Anonymous

    It should be noted that Mike Kenny made an excellent series of comments in this thread.
    Mike, I don’t see why Robin would be skeptical about your proposals, they sound modestly achievable, sound, and useful in filling an information gap.

  • Mike

    I think you misunderstand the nature of dialogue in physics. Certainly certain physicists research predominantly certain ideas, and many if not most view the ideas they research more favorably than those who don’t research those ideas. However, physicists appreciate that it is Nature that decides which theories are correct. If person X’s theory is confirmed, X will NOT claim to have been “right” or to have “known all along,” as if he or she had some special access to truth.

    There is some historical precedent for this. For example, long ago Coleman developed a unified theory of all (non-gravitational) forces, and its simplicity and elegance made it quite compelling. It made a testable prediction about proton decay, which subsequently was shown to be incorrect. This history is described among physicists as “unfortunate” — in the sense that Coleman’s theory could have been correct but it simply wasn’t the way of things — and not as if Coleman somehow lacked insight for making an incorrect prediction.

    Perhaps it helps to clarify that physicists are hoping that the LHC will discern between different *theories*, not the predictions of a given theory. Perhaps at times, a given theory was seen as complicated or not fully understood, and different people decided the theory should predict different outcomes for the same experiment. Then, if the experiment were performed and one were right, the community might view that physicist as the “more insightful” one. But that’s because physicists view the theory as making just one prediction, and understanding that prediction is a matter of understanding the theory. In the present case, we have competing theories, and there is no understanding that can reveal which theory is correct — all you can do is check the results of experiment and see.

  • tndal

    Just finished watching Google Tech Talks presentation by Edward Farhi at

    “Why Physicists Need the LHC”

    Farhi hopes that something (anything, please!) will be found that is not currently predicted by the Standard Model. Otherwise High-energy particle physics in its current form, accurate to 10 decimal places, will be almost completely explained.

    What’s wrong with the Standard Model being correct? It would appear that 10 decimal places accuracy is sufficient to indicate a theory correct and complete.

    High-energy particle physicists are worried that, should the LHC prove to be a $6B boondoggle contributing little or nothing to knowledge, they will be unable to get future funding and will face future unemployment. Farhi admits he is hedging his bets by doing side research in quantum computing (that’s why he’s at Google).

    I hope the LHC finds nothing new. Then research money could be redirected from high-energy particle physics to more productive areas of physics and the other sciences.

  • frelkins

    Is this more along the lines of what you wanted – a prediction on finding the Higgs?

    CERN is losing ground rapidly in the race to discover the elusive Higgs boson, its American rival claims.

    Fermilab say the odds of their Tevatron accelerator finding it first are now 50-50 at worst, and up to 96% at best.