How do you tell whether something is good or bad? Human judgment is surprisingly swayed by contextual cues, rather than by the actual attributes of the thing being judged. As a recent Boston Globe article
This is all true, I think. At the same time, I'm not sure that it undermines my point. What you say shows that we may not care when someone perceives a $5 wine as a $90 wine -- de gustibus non est disputandum, after all -- whereas we do care when the truth of an idea is misperceived.
Right . . . but I think the same sort of cognitive bias can cause our perceptions to change in either case. If I think a $5 wine is a $90 wine, then even though my taste experience might be "valid" in some sense, it's still the case that my perceptions have changed simply because the label changed. So too, if I think a run-of-the-mill idea is insightful just because of the name attached to it, then my perceptions have been altered just because of the label.
It seems to me that there is a difference between the first two examples given by Mr. Buck and his own example about the overvaluing of ideas. What the two examples shows is that you can improve the taste of wine simply by making it more expensive and that you can improve the effectiveness of a pain-killer simply by making it more expensive. In these cases, it's not that people are overvaluing the wine or pain-killer. The wine really does taste better at $90 a bottle, as the scanners confirm. Ditto for the pain-killer. That you can improve the quality or effectiveness of something simply by raising its price might seem strange or spooky, but it doesn't show that they overvalued the items in question.
The value of an idea, by contrast, doesn't increase simply because we find the idea more persuasive if it comes from one person rather than another. The difference is that in the case of wine and placebos, what we want are certain experiences (taste or absence of pain), whereas when it comes to ideas what we want is not the appearance of truth, but actual truth.
As for the example of the Ape and the Art-Dealer, all I think that shows is that the contemporary art scene is full of it.
Irrc at one point the show "20/20" had a kindergarden class draw paintings. Then a group of professional art critics was assembled and presented abstract paintings with half being those of professional artists and the other half of the paintings being those done by the children. The critics were told it was all professional.
Guess what happened...
None of the critics had a clue. They started using fancy words to describe how the artists were using advanced techniques to express inner angst and bla bla...
When one of the critics was praising a child's painting for it's creative use of contrast the man was told it was painted by a six-year-old(I think). He was slightly stumped and blushed a bit but then retorted "They should give the kid a gallery". All the critics became silent once told half the paintings were actually done by a random group of children. Later a critic said jokingly "Art is for rich people who don't know what to do with their money".
The entire art industry is based on mostly hype and reputations.
This reminds me of the Penn and Teller episode where they had a "bottled water tasting" at a fancy restaurant. All the bottles, however, were filled with tap water. Makes for fun watching.
Do you prefer the finest wines, vehicles and art? How do you define the finest? Whatever you may think, you will find this fascinating.SCIENTISTS AT CALTECH and Stanford recently published the results of a peculiar wine tasting. They provided people
I always find that my enjoyment of videogames, and to a lesser extent TV shows, has a lot to do with the reviews I've read. I'm not too focused on overcoming this bias though, since I don't know how to (other than avoiding reviews) and it doesn't seem particularly detrimental to me, since it's a subjective matter.
But we should give pundits lots of credit, and look hard for value in the things they say. Right?
Or maybe we should try to overlook the fact that particular people say particular things at particular places and times, and try to maintain evenhanded skepticism towards all assertions, ideas, and arguments.
This post really seems to be about overvaluing provenance, rather than ideas as such. Provenance seems to play a very great role indeed in how we value things, and this value can't always be explained as simply information as to the inherent qualities of the end product.
For example, one concept I've always found perverse is that of forgery of a work of art, assuming there is no copyright infringement (from here on, assume the 'original master' is long dead). There is an incentive to make paintings that look just like the work of famous painters and pass them off as being *by* the famous painter, because this false provenance will give a higher market value. (It could be a direct copy of a painting, or even just 'in the style of' a given painter - the former may bear a higher risk of detection, as two identical copies of a painting on the market will arouse suspicions.) If the true painter becomes known, the painting suddenly drops in market value, and we might say the owner of the forgery at the time has a right to sue for the loss. But should they sue the painter for making the forgery, or whoever leaked the information that it was a forgery? Arguably there is a good case for the latter, as it is the *knowledge* of the true origin of the painting that has destroyed its market value, not the origin itself. (By similar logic, in some jurisdictions truth and honesty are no defence against an accusation of defamation.)
What exactly are people paying for when they value works by 'famous masters' more highly, regardless of the level of quality or skill evident in the work itself - provenance, or merely the illusion of provenance?
Why an arbitrary scale? Why not a probability? It would be fairly easy to dig up every paper written by academicians who are "famous" in some way, and then compare the error rate to that of papers overall.
On second thought, you really need two scales: one for probability of correctness, and one for usefulness if the idea is correct. Using these, you can get the third scale (expected utility), which is the one most people care about. It's perfectly possible that (say) famous academicians have a lower error rate on every class of idea, but they propose really important ideas more frequently than non-famous academicians, and so their error rate is higher overall.
Regardedness (is that a word?) may very well affect the probability of new ideas being correct, if you actually did out the Bayesian math. However, it is irrelevant to what the importance of an idea is if it's true; you can usually determine an idea's importance just by looking at it. Note that I'm assuming that science will approach truth asymptotically, so an idea is still just as important if (say) a non-famous guy invents it, it gets ignored for twenty years, and then a famous guy invents it independently and it rewrites the textbooks. Evaluating utility-if-true independently from probability may help us to reduce the likelihood of that scenario.
"-Does anyone still believe there's objectivity to wine-tasting?"
So you shouldn’t actually reject an idea just for being highly-regarded. But suppose that, because of the herd effect described above, the most highly-regarded ideas are treated as if they have a value of 85 or 90 — far above their real value.
Well put.
I think such assessment tends to be fairly accurate after a century or so, at least in science; but the highly-regarded new ideas in a current field probably do exhibit your pattern. Some of them are genuine 90s, and will be seen as such a century later, but those are far rarer than ideas treated as 90s.
This isn't news to people like me, given the ~50,000 times I've made a point on a forum, been ignored or ridiculed, and then seen someone else (more respected) later make the exact point afterward and been lauded for his genius.
Ironically, since your post wasn't by Eliezer_Yudkowsky or Robin_Hanson, I was going to ignore it except that I was bored. Go fig :-P
Few more points:
-Does anyone still believe there's objectivity to wine-tasting?
-Don't forget Joshua Bell, or his reaction. ("I didn't get validation in advance! WAH!")
-Neither the bottled water blind test, nor the back pain test will work on me. I've rejected bad bottled water, even when I've exptected it to be good. I've also had back pain for over 10 years, despite taking ~10 different medications I expected to work. If there's money to made from me being correct about these claims, point me in the right direction.
-The fact that an expert can see a monkey's work as art, shows why I don't take academics' opinions on art seriously. I actually have no pre-judgment against the idea that a monkey might one day produce genuine art. I even find it probable. But it will have to do better than abstract expressionism :-P
I read your entire post. I would have skimmed or skipped it if it had been published in my local newspaper. I expect posts on this blog to be more "worthy" than those by a small-town newspaper, and by reading to the end I think I found some value in the article. However, I find it interesting that an article on bias risk, in a blog devoted to overcoming bias, provides me with anecdotal reason to be biased with my reading choices.
Stuart,
Yes, I grok you.
Blackadder --
This is all true, I think. At the same time, I'm not sure that it undermines my point. What you say shows that we may not care when someone perceives a $5 wine as a $90 wine -- de gustibus non est disputandum, after all -- whereas we do care when the truth of an idea is misperceived.
Right . . . but I think the same sort of cognitive bias can cause our perceptions to change in either case. If I think a $5 wine is a $90 wine, then even though my taste experience might be "valid" in some sense, it's still the case that my perceptions have changed simply because the label changed. So too, if I think a run-of-the-mill idea is insightful just because of the name attached to it, then my perceptions have been altered just because of the label.
Am I making any sense?
It seems to me that there is a difference between the first two examples given by Mr. Buck and his own example about the overvaluing of ideas. What the two examples shows is that you can improve the taste of wine simply by making it more expensive and that you can improve the effectiveness of a pain-killer simply by making it more expensive. In these cases, it's not that people are overvaluing the wine or pain-killer. The wine really does taste better at $90 a bottle, as the scanners confirm. Ditto for the pain-killer. That you can improve the quality or effectiveness of something simply by raising its price might seem strange or spooky, but it doesn't show that they overvalued the items in question.
The value of an idea, by contrast, doesn't increase simply because we find the idea more persuasive if it comes from one person rather than another. The difference is that in the case of wine and placebos, what we want are certain experiences (taste or absence of pain), whereas when it comes to ideas what we want is not the appearance of truth, but actual truth.
As for the example of the Ape and the Art-Dealer, all I think that shows is that the contemporary art scene is full of it.
This makes me think of Roald Dahl's wine and art pranks. I have also seen the Penn and Teller episode before.
Irrc at one point the show "20/20" had a kindergarden class draw paintings. Then a group of professional art critics was assembled and presented abstract paintings with half being those of professional artists and the other half of the paintings being those done by the children. The critics were told it was all professional.
Guess what happened...
None of the critics had a clue. They started using fancy words to describe how the artists were using advanced techniques to express inner angst and bla bla...
When one of the critics was praising a child's painting for it's creative use of contrast the man was told it was painted by a six-year-old(I think). He was slightly stumped and blushed a bit but then retorted "They should give the kid a gallery". All the critics became silent once told half the paintings were actually done by a random group of children. Later a critic said jokingly "Art is for rich people who don't know what to do with their money".
The entire art industry is based on mostly hype and reputations.
This reminds me of the Penn and Teller episode where they had a "bottled water tasting" at a fancy restaurant. All the bottles, however, were filled with tap water. Makes for fun watching.
Nothing But The Best
Do you prefer the finest wines, vehicles and art? How do you define the finest? Whatever you may think, you will find this fascinating.SCIENTISTS AT CALTECH and Stanford recently published the results of a peculiar wine tasting. They provided people
I always find that my enjoyment of videogames, and to a lesser extent TV shows, has a lot to do with the reviews I've read. I'm not too focused on overcoming this bias though, since I don't know how to (other than avoiding reviews) and it doesn't seem particularly detrimental to me, since it's a subjective matter.
But we should give pundits lots of credit, and look hard for value in the things they say. Right?
Or maybe we should try to overlook the fact that particular people say particular things at particular places and times, and try to maintain evenhanded skepticism towards all assertions, ideas, and arguments.
That way we avoid devolving into cultists.
This post really seems to be about overvaluing provenance, rather than ideas as such. Provenance seems to play a very great role indeed in how we value things, and this value can't always be explained as simply information as to the inherent qualities of the end product.
For example, one concept I've always found perverse is that of forgery of a work of art, assuming there is no copyright infringement (from here on, assume the 'original master' is long dead). There is an incentive to make paintings that look just like the work of famous painters and pass them off as being *by* the famous painter, because this false provenance will give a higher market value. (It could be a direct copy of a painting, or even just 'in the style of' a given painter - the former may bear a higher risk of detection, as two identical copies of a painting on the market will arouse suspicions.) If the true painter becomes known, the painting suddenly drops in market value, and we might say the owner of the forgery at the time has a right to sue for the loss. But should they sue the painter for making the forgery, or whoever leaked the information that it was a forgery? Arguably there is a good case for the latter, as it is the *knowledge* of the true origin of the painting that has destroyed its market value, not the origin itself. (By similar logic, in some jurisdictions truth and honesty are no defence against an accusation of defamation.)
What exactly are people paying for when they value works by 'famous masters' more highly, regardless of the level of quality or skill evident in the work itself - provenance, or merely the illusion of provenance?
"Put it this way: On a scale of 1 to 100,"
Why an arbitrary scale? Why not a probability? It would be fairly easy to dig up every paper written by academicians who are "famous" in some way, and then compare the error rate to that of papers overall.
On second thought, you really need two scales: one for probability of correctness, and one for usefulness if the idea is correct. Using these, you can get the third scale (expected utility), which is the one most people care about. It's perfectly possible that (say) famous academicians have a lower error rate on every class of idea, but they propose really important ideas more frequently than non-famous academicians, and so their error rate is higher overall.
Regardedness (is that a word?) may very well affect the probability of new ideas being correct, if you actually did out the Bayesian math. However, it is irrelevant to what the importance of an idea is if it's true; you can usually determine an idea's importance just by looking at it. Note that I'm assuming that science will approach truth asymptotically, so an idea is still just as important if (say) a non-famous guy invents it, it gets ignored for twenty years, and then a famous guy invents it independently and it rewrites the textbooks. Evaluating utility-if-true independently from probability may help us to reduce the likelihood of that scenario.
"-Does anyone still believe there's objectivity to wine-tasting?"
No. See http://scienceblogs.com/cor....
So you shouldn’t actually reject an idea just for being highly-regarded. But suppose that, because of the herd effect described above, the most highly-regarded ideas are treated as if they have a value of 85 or 90 — far above their real value.
Well put.
I think such assessment tends to be fairly accurate after a century or so, at least in science; but the highly-regarded new ideas in a current field probably do exhibit your pattern. Some of them are genuine 90s, and will be seen as such a century later, but those are far rarer than ideas treated as 90s.
This isn't news to people like me, given the ~50,000 times I've made a point on a forum, been ignored or ridiculed, and then seen someone else (more respected) later make the exact point afterward and been lauded for his genius.
Ironically, since your post wasn't by Eliezer_Yudkowsky or Robin_Hanson, I was going to ignore it except that I was bored. Go fig :-P
Few more points:
-Does anyone still believe there's objectivity to wine-tasting?
-Don't forget Joshua Bell, or his reaction. ("I didn't get validation in advance! WAH!")
-Neither the bottled water blind test, nor the back pain test will work on me. I've rejected bad bottled water, even when I've exptected it to be good. I've also had back pain for over 10 years, despite taking ~10 different medications I expected to work. If there's money to made from me being correct about these claims, point me in the right direction.
-The fact that an expert can see a monkey's work as art, shows why I don't take academics' opinions on art seriously. I actually have no pre-judgment against the idea that a monkey might one day produce genuine art. I even find it probable. But it will have to do better than abstract expressionism :-P
I read your entire post. I would have skimmed or skipped it if it had been published in my local newspaper. I expect posts on this blog to be more "worthy" than those by a small-town newspaper, and by reading to the end I think I found some value in the article. However, I find it interesting that an article on bias risk, in a blog devoted to overcoming bias, provides me with anecdotal reason to be biased with my reading choices.
Cheers,
David Rotor