Fake Explanations

Once upon a time, there was an instructor who taught physics students.  One day she called them into her class, and showed them a wide, square plate of metal, next to a hot radiator.  The students each put their hand on the plate, and found the side next to the radiator cool, and the distant side warm.  And the instructor said, Why do you think this happens?  Some students guessed convection of air currents, and others guessed strange metals in the plate.  They devised many creative explanations, none stooping so low as to say "I don't know" or "This seems impossible."

And the answer was that before the students entered the room, the instructor turned the plate around.

Consider the student who frantically stammers, "Eh, maybe because of the heat conduction and so?"  I ask: is this answer a proper belief?  The words are easily enough professed - said in a loud, emphatic voice.  But do the words actually control anticipation?

Ponder that innocent little phrase, "because of", which comes before "heat conduction".  Ponder some of the other things we could put after it.  We could say, for example, "Because of phlogiston", or "Because of magic."

"Magic!" you cry.  "That's not a scientific explanation!"  Indeed, the phrases "because of heat conduction" and "because of magic" are readily recognized as belonging to different literary genres.  "Heat conduction" is something that Spock might say on Star Trek, whereas "magic" would be said by Giles in Buffy the Vampire Slayer.

However, as Bayesians, we take no notice of literary genres.  For us, the substance of a model is the control it exerts on anticipation.  If you say "heat conduction", what experience does that lead you to anticipate?  Under normal circumstances, it leads you to anticipate that, if you put your hand on the side of the plate near the radiator, that side will feel warmer than the opposite side.  If "because of heat conduction" can also explain the radiator-adjacent side feeling cooler, then it can explain pretty much anything.

And as we all know by this point (I do hope), if you are equally good at explaining any outcome, you have zero knowledge.  "Because of heat conduction", used in such fashion, is a disguised hypothesis of maximum entropy.  It is anticipation-isomorphic to saying "magic".  It feels like an explanation, but it's not.

Supposed that instead of guessing, we measured the heat of the metal plate at various points and various times.  Seeing a metal plate next to the radiator, we would ordinarily expect the point temperatures to satisfy an equilibrium of the diffusion equation with respect to the boundary conditions imposed by the environment.  You might not know the exact temperature of the first point measured, but after measuring the first points - I'm not physicist enough to know how many would be required - you could take an excellent guess at the rest.

A true master of the art of using numbers to constrain the anticipation of material phenomena - a "physicist" - would take some measurements and say, "This plate was in equilibrium with the environment two and a half minutes ago, turned around, and is now approaching equilibrium again."

The deeper error of the students is not simply that they failed to constrain anticipation.  Their deeper error is that they thought they were doing physics.  They said the phrase "because of", followed by the sort of words Spock might say on Star Trek, and thought they thereby entered the magisterium of science.

Not so.  They simply moved their magic from one literary genre to another.

Comments

[Anonymous Coward]

Well, one difference between "heat conduction" and "phlogiston" is that the former carries some additional information with it - heat conduction is a well-understood mechanism by which energy is transferred from place to place. Maybe it does apply in that situation and maybe it doesn't - in the example given, it doesn't, there's no heat-conduction mechanism to transfer heat from one side to the other - but the fact that there's actually a mechanism behind the words separates it, qualitatively, from an explanation like "phlogiston." It has equations behind it which can then be written down and tested for agreement with reality.

Really, I can quite understand the students... if you say "I don't know" you have a zero percent chance of getting the explanation right. If you say "that seems impossible," then you're guaranteed to get it 100% wrong - since it DID happen, and thus it must be possible. The best course of action in the situation is to think of all the hypotheses you can, and then guess at one of them - whichever one has the highest chance of being right, given what they know about physics.

Now, I certainly hope that the students wouldn't think that by throwing around guesses they're "doing physics" - yes, doing physics would involve taking actual measurements, and I would hope that after taking some measurements of the block over time they would see "oh, this isn't actually at equilibrium like we had all assumed." (Alternatively, if a student took the words and wrote down an actual model of how the air currents or the different metals or the heat conduction could lead to the observations, that would also be "doing physics", though the only end result of it would be to yield a mathematical model which would quickly be easy to proven false by measurements or stability analysis.) But neither of those avenues is open to them when they walk into a classroom and the teacher asks them to "explain this phenomenon."

I think the students would quite happily agree that they haven't given an explanation which is good by any sane measure - it's quite likely that many of them would also agree that they don't actually believe their explanations. But I wouldn't agree that they're being irrational in stating them.

[Eliezer Yudkowsky]

AC, what you're describing here is a severe case of déformation educationnelle.

Really, I can quite understand the students... if you say "I don't know" you have a zero percent chance of getting the explanation right.

If you say "I don't know" you have a zero percent chance of getting a gold star in the idiot damned school system. But it is still the rational thing to say when, in fact, you don't know. You can easily do worse than maximum entropy if you guess at random.

Furthermore, "getting it right" by guessing the verbal phrase the teacher has in mind, even if the school system gives you a gold star for it, does not necessarily mean that you possess any anticipation-controllers. All you got right was a string of words, like guessing the passphrase to the teacher's login.

"Heat conduction" is a verbal phrase which may, for someone who knows the equations, invoke genuinely explanatory equations from memory. And for someone who knows the equations, it should be obvious that the equations do not predict the further side being warmer.

If you don't know the equations, then "heat conduction" is a verbal phrase invoking magic from the Star Trek genre. Even if the teacher says "You're right!", because there were only a limited number of phrases you learned about this semester and one of them had to be "right", you still don't know anything except an arbitrary passphrase.

If you memorized the equations but you didn't apply them, then "heat conduction" still invokes magic - it's not enough to know what symbols to scribble on the final exam, you've got to do the math or it doesn't count.

[Alexis Perrier]

Dear Sir,
Although your article follows the essence of logic, may I point out that Mr Spock, a personal mentor, has never had anything to do with misleading scientific explanations. Mr Spock just happens to live in the future where such things as "plasma conduits" and "warp engines" are common scientific tools.
Yours truly
Alexis Perrier

[ed johnson]

I agree with AC...you're being too hard on the students. I doubt very much they were stating anything with confidence. It's quite possible that some of them didn't really care about understanding physics and were just trying to get the right answer to please the teacher, but others were probably just thinking out loud. Thinking "maybe it's heat conduction" might just be the first step to thinking "no, it can't be heat conduction," or even to realizing "I don't really understand heat conduction," and there is nothing wrong with this train of thought. They were probably "biased" towards the idea that there was some physical principle causing the effect, but that was entirely rational because the professor set them up to believe that.

Great story, though.

[Senthil]

Ed, the student's response may be due to something he needs to unlearn as discussed in the following earlier post:

http://www.overcomingbias.com/2007/07/two-more-things.html

If it's not the case, that is, if he doesn't need to unlearn anything he may still be incorrect in his understanding. In that case, this post tells one of the reasons why he may be incorrect and be aware of it.

I think this is worse compared to the behaviour addressed in the earlier post.

[Stuart Armstrong]

but that was entirely rational because the professor set them up to believe that.

They were rational, but not unbiased. They wanted to maximise their chances of pleasing the prof., not maximise their chances of understanding the world.

I think this teaching approach was great, and I might use something similar myself (there are mathematical equivalents of the above situation). Learning science means that you have to learn a boatload of facts, and learn the scientific method. Since the boatload of facts has to be accepted without question (for the whole of your early career), this undermines the teaching of the method (when it is taught at all). A few sessions like this (properly exploited by the instructor) would do a world of good.

[Hopefully Anonymous]

Great post, Eliezer, and I agree with Stuart. There should be no valor in stating an uncertain guess as a certain statement -one should at least express one's level of uncertainty.

[Hopefully Anonymous]

Incidentally, this is an area where legal instruction is superior to scientific instruction at the graduate/pre-thesis level.

[conchis]

"They wanted to maximise their chances of pleasing the prof., not maximise their chances of understanding the world."

I don't know that I buy this. If the students make a guess that's wrong, one would expect that to kickstart a process of the professor helping them to understand why it's wrong. (Student: "Um... because of heat conduction?" Teacher: "OK, what does heat conduction suggest should happen in this situation?"...) This seems more likely to result in learning than just sitting there and saying "I don't know". If anything, I think it's often a bigger problem from a learning perspective, when people are too afraid of being wrong to put out tentative ideas.

"I don't know" is a rational response to this situation if you are sure enough of your understanding of all the potential principles involved that you know they can't explain the phenomenon (and you don't happen to guess that the professor is messing with you). But it's fairly clear the students aren't in that situation, so starting to generate hypotheses about what's going on seems perfectly sensible. Of course, they should be actual hypotheses, and Eliezer's perfectly right that "because of heat conduction", if offered as an actual explanation, isn't an hypothesis as much as a cop out. But if it's a starting point, rather than an endpoint, then that seems perfectly reasonable.

In short, the problem isn't that they're guessing. It's if their guesses aren't actually saying anything, but they think that they are. (And I think Eliezer's admonition to just say "I don't know" conflates these two problems.)

[Byrne]

Is there any way to set up a classroom (or an educational system) so that these students would get the right answer? Alternatively, is it even desirable?

If you teach students to think this way, you're saying "The world is governed by comprehensible scientific laws -- which is irrelevant, because people are constantly screwing with you." This experiment might be useful in a physics class for lawyers (who would probably catch on) or conspiracy theorists (who would, at least, have more entertaining hypotheses).

A compromise might be for the teacher to not just ask for an explanation, but ask for a testable explanation, and reward people for coming up with a theory they can falsify. The sad truth is that there's one more group that would figure out the answer right away: Creationists.

[Eliezer Yudkowsky]

I should note that I read about this scenario in the Canonical List of Science Jokes but I have no idea whether it was a note from someone's experience. If anyone tries this, I'd love to know the result - my guess is that in real life, at least one student in the class would guess it, which is why I'd suggest having students write down their best suggestions on paper; followed by the teacher asking "How could you falsify your theory?" and writing that down as well.

Explaining things by magic has been the default state of human existence for far longer than science ever existed. Anyone using fancy words must be assumed to be invoking magic by default.

The training of a rationalist must be strict! No human can be unfair enough; you have to match swords against Nature to develop the requisite skills.

[Bob]

I would like to re-emphasize Eliezer's point that "I don't know" (not an uneducated guess) is the proper answer to any question where, in fact, a student (or person in general) does not know the right answer, with the addition of "but I will find out." On my exams a (fully) incorrect answer gets zero points while "I don't know but I will find out" gets one-half credit. You still fail if you don't know anything, but at least you are not in an idiot damned school system. Rewarding students for data dumps when they don't know an answer cannot be healthy. Or maybe I'm just biased because my approach makes exam grading significantly easier....

[conchis]

Bob, Unless guessing is part of finding out. (This clearly isn't the case in an exam, but often is in a classroom situation.)

[Robin Hanson]

Eliezer, I hope you are considering writing a book based on this excellent series of essays you have been writing.

[Eliezer Yudkowsky]

Conchis, the problem is guessing passphrases instead of anticipation-controllers.

Robin, I am indeed considering it, it will depend on how much raw material I can generate.

[Shakespeare's Fool]

A student who said it was done by magic would,
of course, have been correct. Because it was
done by magic.

The teacher moved the plate when the audience
wasn't looking. That is one of the ways
magicians perform their tricks.

If they had used words such as "supernatural,"
"miracle," or "paranormal," then they would
not have been discussing physics.

But good magicians are the best practical
physicists.

[Senthil]

Eliezer, are you also considering giving free copies of the book to people who frequent this blog? :-)

[conchis]

Eliezer, Agreed. (That was my original point.)

[beowulflovesgrendel]

The larger experiment seems to me to be the teacher's looking for someone with an answer to the ENTIRE 'experiment' which includes a 'false' set up. This isn't about 'physics,' it's about overall discernment, much the same way a truly observant participant will 'see through a magic trick,' no mean feat. So, for me, "I don't know" is the only honest and complete answer. It denotes an empty glass which (at least) can be filled and restricts that answwer to a particular observer and does not make 'unknowable" a universal state. Answers which are "data dumps" lull the mind into not questing further. You cannot 'wake up' until you know you have been 'asleep.' AND THEN, there is power imbalance. In a really optimal teaching environment, any 'student' (regardless of age, sex etc.) could forthrightly ask of any 'teacher,' "Is this a set up?" How rare is that?

[beowulflovesgrendel]

regretting the above typos

[ron]

I find it difficult to believe that none of the students would have guessed that the plate was turned around.