James Surowiecki starts his book The Wisdom of Crowds telling how Francis Galton in 1907 used a crowd to guess an ox’s weight: Galton borrowed the tickets from the organizers and ran a series of statistical tests on them Galton arranged the guesses (which totaled 787 in all, after he had to discard thirteen because they were illegible) in order from highest to lowest and graphed them to see if they would form a bell curve. Then, among other things, he added all the contestants’ estimates, and calculated the mean of the group’s guesses. That number represented, you could say, the collective wisdom of the Plymouth crowd. If the crowd were a single person, that was how much it would have guessed the ox weighed. … The crowd has guessed that the ox, after it had been slaughtered and dressed would weigh 1,197 pounds. After it had been slaughtered and dressed, the ox weighted 1,198.
An apology! Very good. I commend Mr. Levy for making one. A scholar should always make it clear after he has made a mistake, of which all of us make many--- though often they're never cleared up.
Are Levy and Peart the pair who have been smearing the Galtonians for years? Jeez, you'd think they would have something better to do with their time...
Galton was 85 years old when he did the guess-the-weight study and realized that his preconception was completely wrong. So, rather than forget about it and take a nap, he reported his new finding to Nature, which became the starting point for the "Wisdom of Crowds" school of thought that James S. wrote about.
Galton was a great scientist and a great man.
For a less tendentious assessment of Galton's many achievements (and fewer, but still real, shortcomings), see Jim Holt's article in The New Yorker:
I have had time to reflect and now Iwould like to offer a more detailed personal apology than what we've jointly posted before. When I failed to find Galton's mean, in spite of your sufficient directions, I should have asked you directly for help. From these two failures of mine, and because Sandy trusted my work, we were led to the wrong conclusion that your account of Galton's mean was false instead of the right conclusion that your account was simply different than our accounts of Galton's median. If theaccounts are merely different then we have many ways of asking which of the two estimators one might prefer. We began that helpful exercise. We did not stop there. When we said that your account was false, and asked a rhetorical question of how this came to be, we called into question my own intentions. We also wrongly called into question the care which scholars took in citing your work.
One paragraph which will go into the next version is this:
One of Galton's defenses for the sample median as the vox populi was it that bounds the influence of any individual voter. Replication and checking of the work of experts may be a way to bound the influence of experts. It is important for reader to know that in an earlier version we denied the existence of Galton's mean. This emphasizes the importance of replication and competition precisely to bound the influence of such error
Here's what we are prepared to defend :
The majority-rule context of Galton's publications is lost when the sample median, upon which Galton put such stress, is no longer reported
To finish, Levy and Peart insist that their really important point still stands, which is that "When people quote Galton through Surowiecki, they tell Surowiecki's tale, not Galton's," and that this is a problem because Galton's thinking is being misrepresented. But as I said earlier, "The Wisdom of Crowds" was not intended to be a discussion of Francis Galton's opinions on what's the best method to capture group judgment, nor, as far as I know, has anyone who's "Surowiecki's tale" used the Galton example since used it to analyze Galton's opinions. People aren't quoting the Galton story because they're interested in what Galton himself thought about the median vs. mean. They're quoting it because they're interested in the bigger idea, which is that group judgments (and this is true whether you use the median, the mean, or a method like parimutuel markets) are often exceptionally accurate. Levy and Peart have constructed a straw man -- and, in this case, a straw man based on a falsehood -- and then tried to knock it down.
Robin writes: "it is ironic that Galton made quite an effort to emphasize and prefer the median, in part because the data did not look like a bell curve, while your retelling focuses on him calculating a mean after checking for a bell curve." What's ironic about this? He did check for a bell curve, and he did calculate the mean. It's the data themselves, not Galton's interpretation of them, that I was writing about. (If he hadn't calculated the mean, I would have happily told the story with the median, since it was also remarkably accurate, and demonstrated the same point about the wisdom of crowds.)
Finally, on the substantive question, Robin (and Levy and Peart) seem to think that because the distribution of guesses wasn't normal, that makes using the mean a mistake. But this is precisely what's so interesting: if the group is large enough, even if the distribution isn't normal, the mean of a group's guesses is nonetheless often exceptionally good.
I appreciate Levy and Peart admitting their mistake. But they seem not to recognize that their mistake undermines the critique that's at the center of their paper. Their paper, they write, is about the misconstruing of Galton's experiment. "A key question," they write, "is whether the tale was changed deliberately (falsified) or whether, not knowing the truth, the retold (and different) tale was passed on unwittingly." But the account of Galton's experiment was not changed deliberately and was not falsified. It was recounted accurately. Levy and Peart want to use my retelling of the Galton story as evidence of how "experts pass along false information(wittingly or unwittingly) [and] become part of a process by which errors are diffused." But there's no false information here, and no diffusion of errors, which rather demolishes their thesis. If they really want to write a paper about how "experts" pass along false information, they'd be better off using themselves as Exhibit A, and tell the story of how they managed to publish such incredibly shoddy work and have prominent economists uncritically link to it.
It looks like there wasn't much variance in the individual estimates that Galton checked! Perhaps people really are unbiased weighers, with experience at county fairs. (I would previously have agreed with Tom McCabe.) Post facto rationalization: Perhaps this is because if people systematically overestimate or underestimate at county fairs, they can correct this with relatively little experience: "Gee, that's the third time in a row my estimate was too high."
I do not think Galton's experiment is valid; there is far too little systematic bias, given the undoubtedly huge variance in the individual estimates, and the difficulty of trying to weigh an object by sight. I defy the data.
James, I've put a line through the clearly incorrect sentence of mine re Pearson, and added a [clarifier] to my first sentence. I also edited the added in the direction of Eliezer's suggestion, and added a question mark to the title. That's about as far as I think it reasonable to go to correct the post.
I agree Levy and Peart were wrong about Pearson 1924 having first calculated the mean. But I do think they have a point that it is ironic that Galton made quite an effort to emphasize and prefer the median, in part because the data did not look like a bell curve, while your retelling focuses on him calculating a mean after checking for a bell curve.
There's no reason for debate here. Levy and Peart say "Pearson’s retelling of the ox judging tale apparently served as a starting point for the 2004 popular account of the modern economics of information aggregation, James Surowieki’s Wisdom of Crowds." It wasn't the starting point. The starting point was Galton's own experiment, and his own reporting of the mean in "The Ballot Box." Robin writes: "Galton did not even bother to calculate a mean." He did calculate it, and he did report it. This fact shouldn't be listed as an "addendum" to the original post. The original post should be rewritten completely -- perhaps along the lines of "Surowiecki and Galton disagree about which estimate is a better representation of group judgment" rather than "Author Misreads Expert" -- or else scrapped.
An apology! Very good. I commend Mr. Levy for making one. A scholar should always make it clear after he has made a mistake, of which all of us make many--- though often they're never cleared up.
What a fiasco...
Are Levy and Peart the pair who have been smearing the Galtonians for years? Jeez, you'd think they would have something better to do with their time...
Galton was 85 years old when he did the guess-the-weight study and realized that his preconception was completely wrong. So, rather than forget about it and take a nap, he reported his new finding to Nature, which became the starting point for the "Wisdom of Crowds" school of thought that James S. wrote about.
Galton was a great scientist and a great man.
For a less tendentious assessment of Galton's many achievements (and fewer, but still real, shortcomings), see Jim Holt's article in The New Yorker:
http://www.newyorker.com/ar...
Levy and Peart need to do some soul-searching about why they are so biased against the Galtonians.
Dear James,
I have had time to reflect and now Iwould like to offer a more detailed personal apology than what we've jointly posted before. When I failed to find Galton's mean, in spite of your sufficient directions, I should have asked you directly for help. From these two failures of mine, and because Sandy trusted my work, we were led to the wrong conclusion that your account of Galton's mean was false instead of the right conclusion that your account was simply different than our accounts of Galton's median. If theaccounts are merely different then we have many ways of asking which of the two estimators one might prefer. We began that helpful exercise. We did not stop there. When we said that your account was false, and asked a rhetorical question of how this came to be, we called into question my own intentions. We also wrongly called into question the care which scholars took in citing your work.
For all this, again, I offer a personal apology.
Best wishes,
David
Why can't all arguments be like this? Congratulations all 'round.
The paper has been taken down at Adam Smith Lives for rethinking. We offer our apologies to James Surowiecki.
http://adamsmithlives.blogs...
One paragraph which will go into the next version is this:
One of Galton's defenses for the sample median as the vox populi was it that bounds the influence of any individual voter. Replication and checking of the work of experts may be a way to bound the influence of experts. It is important for reader to know that in an earlier version we denied the existence of Galton's mean. This emphasizes the importance of replication and competition precisely to bound the influence of such error
Here's what we are prepared to defend :
The majority-rule context of Galton's publications is lost when the sample median, upon which Galton put such stress, is no longer reported
David Levy
To finish, Levy and Peart insist that their really important point still stands, which is that "When people quote Galton through Surowiecki, they tell Surowiecki's tale, not Galton's," and that this is a problem because Galton's thinking is being misrepresented. But as I said earlier, "The Wisdom of Crowds" was not intended to be a discussion of Francis Galton's opinions on what's the best method to capture group judgment, nor, as far as I know, has anyone who's "Surowiecki's tale" used the Galton example since used it to analyze Galton's opinions. People aren't quoting the Galton story because they're interested in what Galton himself thought about the median vs. mean. They're quoting it because they're interested in the bigger idea, which is that group judgments (and this is true whether you use the median, the mean, or a method like parimutuel markets) are often exceptionally accurate. Levy and Peart have constructed a straw man -- and, in this case, a straw man based on a falsehood -- and then tried to knock it down.
Robin writes: "it is ironic that Galton made quite an effort to emphasize and prefer the median, in part because the data did not look like a bell curve, while your retelling focuses on him calculating a mean after checking for a bell curve." What's ironic about this? He did check for a bell curve, and he did calculate the mean. It's the data themselves, not Galton's interpretation of them, that I was writing about. (If he hadn't calculated the mean, I would have happily told the story with the median, since it was also remarkably accurate, and demonstrated the same point about the wisdom of crowds.)
Finally, on the substantive question, Robin (and Levy and Peart) seem to think that because the distribution of guesses wasn't normal, that makes using the mean a mistake. But this is precisely what's so interesting: if the group is large enough, even if the distribution isn't normal, the mean of a group's guesses is nonetheless often exceptionally good.
I appreciate Levy and Peart admitting their mistake. But they seem not to recognize that their mistake undermines the critique that's at the center of their paper. Their paper, they write, is about the misconstruing of Galton's experiment. "A key question," they write, "is whether the tale was changed deliberately (falsified) or whether, not knowing the truth, the retold (and different) tale was passed on unwittingly." But the account of Galton's experiment was not changed deliberately and was not falsified. It was recounted accurately. Levy and Peart want to use my retelling of the Galton story as evidence of how "experts pass along false information(wittingly or unwittingly) [and] become part of a process by which errors are diffused." But there's no false information here, and no diffusion of errors, which rather demolishes their thesis. If they really want to write a paper about how "experts" pass along false information, they'd be better off using themselves as Exhibit A, and tell the story of how they managed to publish such incredibly shoddy work and have prominent economists uncritically link to it.
Surowiecki's Mistake?
In my book, I reference Surowiecki's "guess-the-weight-of-an-ox" anecdote. My colleague David Levy and his co-author Sandra Peart show that isn't...
It looks like there wasn't much variance in the individual estimates that Galton checked! Perhaps people really are unbiased weighers, with experience at county fairs. (I would previously have agreed with Tom McCabe.) Post facto rationalization: Perhaps this is because if people systematically overestimate or underestimate at county fairs, they can correct this with relatively little experience: "Gee, that's the third time in a row my estimate was too high."
A long Levy and Peart response is now added to the post.
Thank you for the opportunity to learn about the power and subtlety of indignation.
I do not think Galton's experiment is valid; there is far too little systematic bias, given the undoubtedly huge variance in the individual estimates, and the difficulty of trying to weigh an object by sight. I defy the data.
James, I've put a line through the clearly incorrect sentence of mine re Pearson, and added a [clarifier] to my first sentence. I also edited the added in the direction of Eliezer's suggestion, and added a question mark to the title. That's about as far as I think it reasonable to go to correct the post.
I agree Levy and Peart were wrong about Pearson 1924 having first calculated the mean. But I do think they have a point that it is ironic that Galton made quite an effort to emphasize and prefer the median, in part because the data did not look like a bell curve, while your retelling focuses on him calculating a mean after checking for a bell curve.
Perhaps the addendum could read something like "James Surowiecki says this is completely wrong: Etc."
Here are the links for the letter from Galton, where he reports the mean:
http://galton.org/cgi-bin/s...http://galton.org/cgi-bin/s...
There's no reason for debate here. Levy and Peart say "Pearson’s retelling of the ox judging tale apparently served as a starting point for the 2004 popular account of the modern economics of information aggregation, James Surowieki’s Wisdom of Crowds." It wasn't the starting point. The starting point was Galton's own experiment, and his own reporting of the mean in "The Ballot Box." Robin writes: "Galton did not even bother to calculate a mean." He did calculate it, and he did report it. This fact shouldn't be listed as an "addendum" to the original post. The original post should be rewritten completely -- perhaps along the lines of "Surowiecki and Galton disagree about which estimate is a better representation of group judgment" rather than "Author Misreads Expert" -- or else scrapped.
Here is the link for the facsimile of the Galton article.http://galton.org/essays/19...