Discover more from Overcoming Bias
How does the distribution of truth compare to the distribution of opinion? That is, consider some spectrum of possible answers, like the point difference in a game, or the sea level rise in the next century. On each such spectrum we could get a distribution of (point-estimate) opinions, and in the end a truth. So in each such case we could ask for truth’s opinion-rank: what fraction of opinions were less than the truth? For example, if 30% of estimates were below the truth (and 70% above), the opinion-rank of truth was 30%.
If we look at lots of cases in some topic area, we should be able to collect a distribution for truth’s opinion-rank, and so answer the interesting question: in this topic area, does the truth tend to be in the middle or the tails of the opinion distribution? That is, if truth usually has an opinion rank between 40% and 60%, then in a sense the middle conformist people are usually right. But if the opinion-rank of truth is usually below 10% or above 90%, then in a sense the extremists are usually right.
Now the sense in which extremists might be more right is not a sense of a better median or expected value; surely those tend to be near the middle of the distribution of opinion. But as I explained yesterday, we might be rewarded for opinions closer to the truth than other opinions. If truth tended to be in the opinion tails, extremists would on average get more such rewards, while if the truth tended to be in the middle, the conformists would get more rewards. (Other social rewards, for conforming or for showing originality and daring, would probably explain the deviation.)
Extremists could also be more right in the sense of providing more useful explorations of points of view. That is, if the choice of a point estimate was the choice to explore the implications of possible scenarios near that point estimate, we might want the fraction of people who explored each scenario to be proportional to its probability, as well as to the difficulty and importance of thinking about that scenario, if it were true.
So, does truth tend to be found more in the middle hump of opinion, or in the extreme tails? Someone out there must have data that can shed light on this.
Is Truth in the Hump or the Tails?
Douglas Knight,
"I am not aware of any story of disappointment with a proof of a conjecture that did not involve incomprehension of the proof"
De Branges' proof of the Bieberbach conjecture left some people cold because it rested on a magic trick taken out of a hat.
"I am not aware of anyone who groaned at Perelman's solution, except maybe Yau"
Why should Yau be upset? It was Yau's approach that Perelman pushed through. The folks that groaned were the topologists: here is the conjecture that has made their field being solved by PDE methods, no wonder they were not pleased. (By the way, I'm not trying to put down Perelman or anybody else, but if we don't discuss specific examples it's hard to make any progress).
"Your claim that people would be disappointed in a spectral proof of the Riemann hypothesis is far more absurd".
I don't understand why you think it's absurd.
Finally, I'm too ignorant about P=NP to be able to say anything worthwhile, so I'll shut up on that subject.
PS: This article in today's arxiv (by the recent fields medalist Tao) gives a good description of the various different truths in mathematics and how they come about (quite uexpectedly I would say).
http://www.arxiv.org/abs/ma...
Before I go off on Calca's tangent, complaining about his examples, I will give an example on the difficulty of observing conjectures that are deeply held. One does know that everyone was shocked by IP=PSPACE, but one can only tell that people conjectured a strict inclusion by the difficulty in publishing the inclusion of IP in PSPACE. Of course, I do know this, and I know it because the theorem is celebrated, and that is because, per Calca, it was entirely unexpected and changed the perspective. And they tell the story to demonstrate this. Was the insolubility by radicals of the quintic shocking 200 years ago? I have no idea.
It is true that mathematicians care about how conjectures are proved, and what those methods give more generally. This is the main complaint about extremely computer-aided proofs, like the four color theorem. There is the story of Grothendieck's disappointment with Deligne's proof of the Weil conjectures. Aside from that joke, I am not aware of any story of disappointment with a proof of a conjecture that did not involve incomprehension of the proof (which happens not too rarely with proofs by humans).
I am not aware of anyone who groaned at Perelman's solution, except maybe Yau. Perelman didn't introduce any radically new concepts, but he showed that the Gromov-Hausdorff degeneration techniques are very useful in the context of Ricci flow. It is expected that this, and his other improvements to Ricci flow, will have application to Ricci flow in higher dimensions, particularly on Kaehler manifolds. I suspect that with equal distance from Wiles, you would describe his work as merely pushing through an established program.
Your claim that people would be disappointed in a spectral proof of the Riemann hypothesis is far more absurd. My impression is that the spectral approach is more an idle remark than an established program. Connes does have a specific operator (although I'm not sure it quite fits in the Hilbert-Polya picture) and I imagine that his conjectures about it would have other consequences than the Riemann hypothesis. Certainly, he claims that his perspective has other applications to number theory, but I don't know how broadly one must interpret "perspective."