I doubt that it’s important to read any particular book before reading Causality, but it does require a good deal of comfort with basic statistical theory and with the way math is taught in college level textbooks.

The evidence-based-medicine movement shows signs of overemphasizing statistical knowledge.

The parachute example is a good illustration of the superiority of causal knowledge because the reader knows that falling out of an airplane causes injury because of the velocity with which the person hits the ground. In the vocabulary developed by Pearl and his colleagues, the velocity at which the person hits the ground screens off the effect falling out of the airplane has on the injury.

Eliezer has written that the reader should read Pearl’s Probabilistic Reasoning in Intelligent Systems before attempting Pearl’s Causality book, and has advised students of AI to read Tom Mitchell’s Machine Learning before reading Probabilistic Reasoning in Intelligent Systems.

Robin, it’s unclear what observable outcomes from tin foil hats we should be testing for. For sufficiently broad meanings of tin foil hat, near the end of this post is advice to “Wrap your rooms in aluminum foil” for protection against EMP that ought to be studied.

The authors further imagine that the treatment group and the control group have to be similar in size, that is, the randomisation has to be fifty-fifty. This is completely wrong.

If you wanted to set up a randomised trial of parachute effectiveness you could arrange that 99.9999% of the jumpers were assigned to the treatment group and 0.0001% were assigned to the control group. After 10 million jumps, you would have 10 splattered controls and a statistically significance result.

The biggest difficulty in arranging this is in distinguishing the experiment from what sports parachutests do anyway. When you jump out of the plane and pull the ripcord it is like rolling a dice with a 1 in 250 chance of your main shute failing. Whoops! You have become a candidate. You cut away and try the reserve. Reserves are more reliable. Very roughly one time in 4000 you find yourself assigned to the control group. Good luck. American sports parachutests are basically repeating this randomised controlled trial every four years.

OK no-one is writing this up in the BMJ, so it doesn’t have the red ribbons and gold braid of a randomised controlled trial, but it does have the substance. There is a control group and assignment is randomised. A RCT purist can put his hand on his heart and say: My belief in parachutes is evidence based.

There is another problem with the article. The authors are annoyed at being asked to provide evidence in the form of RCT results for treatments in which they have faith. So they write a sarcastic article. The key ingredient is a treatment which is accepted even though there is no RCT testifying to its effectiveness. Notice the role of controversy in their article.

If a controversial treatment, whose efficacy is doubted, is given a free pass, and allowed into the repetoire of approved treatments without a RCT, the authors can say “what is sauce for the goose is sauce for the gander. Our favourite treatment should also be exempted from the RCT requirement even though it too is controversial.”

Notice the direction of the implied comparison. If the treatments that are exempted from the requirement for RCT evidence are more controversial than their favourite treatment, then their argument is correspondingly strong. They can say “you shouldn’t be bugging us for evidence, you should be focusing on those over there whose anecdotes are thinner and less plausible than ours”

Keep your eye on the direction of the comparison. If they start talking about less controversial treatments, their argument is correspondingly weaker. They might object to demands for RCT evidence to support their favourite treatment, but there is something quite natural about the EBM Nazis coming for the less convincing treatments first.

If they were doing a straight presentation this would be obvious. You show up the hypocrisy of the EBM Nazis by drawing attention to the free passes they give to treatments that might be no good and they are all the more embarrassed if the treatment is especially weak. Unfortunately the authors try to be sarcastic. So they turn things around. Their logic skills are not up to the job and they get the logic of the comparision the wrong way round. Yes, going after a treatment whose efficacy no-one doubts is good comedy, but the more certain the treatment they ridicule, the weaker their argument.

The probability of surviving a sky jump with a working parachute is greater than 99%. The probability of surviving without a parachute is less than 1%. For an effect of that magnitude, you would only need a sample size of six or seven to achieve statistical rigor, while I would personally be satisfied after watching three people with parachutes and with vehicle (a parachute that fails to deploy) jump out of airplanes. If you’ve ever watched TV, you have enough data.