Just Enough Expertise?

Karl Sabbagh on this year’s Edge question:

I used to believe that there were experts and non-experts and that, on the whole, the judgment of experts is more accurate, more valid, and more correct than my own judgment. But over the years, thinking – and I should add, experience – has changed my mind. What experts have that I don’t are knowledge and experience in some specialized area. What, as a class, they don’t have any more than I do is the skills of judgment, rational thinking and wisdom. … Most of us confuse expertise with judgment. …

As a result of changing my mind about this, I now view the judgments of others, however distinguished or expert they are, as no more valid than my own. If someone who is a ‘specialist’ in the field disagrees with me about a book idea, the solution to the Middle East problems, the non-existence of the paranormal or nuclear power, I am now entirely comfortable with the disagreement because I know I’m just as likely to be right as they are.

I’m confident that, as an author and television producer, Karl Sabbagh frequently identifies others he thinks less likely than him to be right.  For example, he probably rejects most book or TV show concepts proposed by ordinary people, justifiably pointing to his superior experience and success in such ventures.  Thus Sabbagh thinks that on most topics he happens to have about as much expertise as is useful; less expertise than his hurts your accuracy, but more expertise than his doesn’t help your accuracy.

I think this same “I have the max useful expertise” view is shared by the typical internet “troll”:

A person who posts incendiary comments with the express purpose of provoking an argument.

Trolls are said to be distinguished by:

  • a lack of buy-in to the list philosophy or values
  • generally low level of activity, with sudden spurts of interaction …
  • a mixture of friendly posts with a confrontational style of interaction
  • the use of provocative language and sweeping generalizations …
  • a lack of in-depth understanding of the topic

This seems to me the classic arrogant young man, who dismisses “smaller” opponents as unworthy, but doubts “bigger” ones could beat him.  He talks tough and swaggers around looking for any easy fight with an audience.  And unfortunately non-expert trolls, and their non-expert audiences, often simply cannot see when their arguments are “beaten” by experts.  As Sean Carroll says:

There is also an unhealthy brand of skepticism, proceeding from ignorance rather than expertise, which insists that any consensus must flow from a reluctance to face up to the truth, rather than an appreciation of the evidence.  It’s that kind of skepticism that keeps showing up in my email.  Unsolicited.  Heresy is more romantic than orthodoxy.  … Many casual heretics can’t be bothered with all the detailed theoretical arguments and experimental tests that support the models they hope to overthrow.

You need non-trivial evidence to justifiably conclude you have just enough expertise.

GD Star Rating
loading...
Tagged as:
Trackback URL:
  • http://www.satisfice.com/blog James Bach

    “Trolls are said to be…”

    Said to be by whom? People who fancy themselves *not* to be trolls? I doubt that the “trolls” would describe themselves that way. Analyze, for a moment, all the biases that are probably operating in the mind of someone who feels that it’s appropriate to label someone else a “troll”…

    A problem with a label like that is that even if there was a way to objectively bestow the label, it’s mistaking a behavior for a nature.

    I maintain several online forums. All of them are moderated. That way people who misbehave don’t get published– except on their own forums.

  • http://profile.typekey.com/Unholysmoke/ Ben Jones

    Robin,

    I believe expertize is a (little-used) verb, i.e. ‘to expertize in a field.’

    Expertise is the noun, even in the States.

    Good post!

  • Greg

    James Bach,

    Oh come now, you’re just being silly. Look:

    “Criminals are said to be…”

    ‘Said to be by whom? People who fancy themselves *not* to be criminals? I doubt that the “criminals” would describe themselves that way. Analyze, for a moment, all the biases that are probably operating in the mind of someone who feels that it’s appropriate to label someone else a “criminal”…’

    We have a set of objective criteria which, if observed, make someone a troll. Or, if you prefer, one who is engaging in trollish behaviour, but the distinction is more than a little odious. Now quit it before I call you a troll.

  • http://rolfnelson.blogspot.com Rolf Nelson

    A similar bias is believing you have “Just Enough Intelligence”. I regularly discount the views of people I personally judge to be less intelligent than me (by some vague criteria), yet not once have I written a letter to Marilyn vos Savant and asked her what her political views are, with the intent of discarding my views and replacing them with hers.

  • Caledonian

    We have a set of objective criteria which, if observed, make someone a troll.

    No, the objective criteria you have are associated with trolls. ‘If A, then B’; ‘B’; there is not enough data to conclude ‘A’.

    The criteria that define trolls involve their intentions, which are difficult at best to determine and people have a natural tendency to leap to conclusions about.

  • Phil

    In the spirit of intrepreting an argument in the way that gives the arguer the most benefit of the doubt, let me rephrase Sabbagh’s point this way:

    1. There are two requirements for being correct: background knowledge, and the logic to apply that knowledge.

    2. Many “experts” have the first, but not the second.

    3. I [Sabbagh] have above-average capability to reason logically. Therefore, knowing nothing else about the expert, it is likely that my reasoning is better than the expert’s.

    4. Therefore, my opinion, at least on subjects in which logic and judgement is important, is at least as good as a random expert’s.

    Sabbagh makes more sense this way. He would have to acknowledge, though, that a certain specific *non-random* expert might be worth listening to.

  • http://lonelygenius.wordpress.com LG

    I think this discussion has to framed more precisely to be useful: “Just enough expertise for _____”?

    Of course it’s not sensible to generalize by saying that expertise is useless, but it is worth noting that expertise in one area does not translate to results in all areas.

    A trivial example is being lost in a desert with an expert… psychologist? The more salient example is an expert whose domain seems to overlap with the situation, but in fact does not; an expert programmer is so good he is promoted to be a manager. It seems to make some sense from the inside looking out, but the reality is that expert programmers are not more likely to know how to manage people or do business just by virtue of their expertise.

    As you said, Sabbagh seems to have generalized his probably true observation that experts are no more likely to choose a winning book or show idea than he is. That’s the result of a domain expertise.

    What he missed is that expertise implies knowledge on a specialized range of subjects which may not have a bearing on whatever the actual desired outcome is — hence, “Just enough expertise _for what_?”

  • tcpkac

    Sabbagh overstates his case. He starts by making the point that expertise is not enough, it needs to be complemented by ‘judgement’ and ‘wisdom’, by which I assume he means good general purpose heuristics on the one hand and breadth of vision on the other.
    Nothing much to argue with so far.
    When he goes on to throw the baby out with the bath water, and imply that judgement and wisdom are all he needs, well, I guess he just got carried away.

  • Shane

    Am I reading a different article than the rest of you? The point isn’t that expertise doesn’t exist, or is useless. The point is that expertise is defined wrt something particular; within that realm you are an expert, and your opinion is worth more. Outside that realm you are not, and it isn’t, or at least, not because of your expertise.

    I think this is a particularly interesting question wrt to the Democratic primary: I’m inclined to think that most of the time, the talk of “experience”, as Clinton uses it, is misleading, and inclined to believe that judgment, as Obama purports himself to have, is one of the very small number of qualities that can actually make a difference in a president. (Really, though, “judgment” is probably just a convenient shorthand for “listening to a wide range of opinions and doing what seems sensible, instead of what is orthodox or traditional.”) But is there any reason to think this?

    I can anticipate a number of wide-ranging arguments on this topic, mostly of the form “there’s no such thing as judgment.” I am mostly sympathetic to this argument, but I also know a lot of shitheads, who bring poor decisions to most endeavors, and people who, on the surface at least, seem to make good decisions repeatedly. So I don’t know.

    BTW, Rolf Nelson: very interesting point, and nice illustration. I’ve saved it for future pondering.

  • Psychohistorian

    That doesn’t seem to be an accurate description of a troll. Trolls create conflict for its own sake. It has nothing to do with their understanding; an effective troll may make a post he knows to be completely false, so as to start an ongoing argument. Making a vague argument and avoiding the specifics of a theory are deliberate calculations; a troll needs to be just intelligent enough to look serious, but not intelligent enough to understand why he’s wrong or make an easily falsifiable claim. That, and broad generalizations tend to make people generate long rants in response. This is, generally, what a troll is aiming to do; start a big, elaborate, and very passionate argument.

    That said, it’s a good point about how people tend to think there’s a binary function of qualified or not qualified, so if they’re qualified they’re on equal footing with everyone else who’s qualified. But I think the internet troll example is not terribly relevant.

  • http://hanson.gmu.edu Robin Hanson

    Ben, thanks, I’ve fixed the spello.

    Rolf, yes, similar biases hold for any X correlated with accuracy; people think that however much X they have is just enough, and more isn’t helpful.

    Phil, under your interpretation he thinks he has just enough judgment; otherwise he’d think people with more judgment that he are right more often.

  • Silas

    For example, he probably rejects most book or TV show concepts proposed by ordinary people, justifiably pointing to his superior experience and success in such ventures

    This justification is not very rigorous. There is a huge number of factors influencing the success of a movie or TV show. Viewers could simply like one actor, or the special effects (Star Wars prequels, I’m looking in your general direction). Further, most successful concepts do in fact sound stupid, even to him. To test whether he really is good, you would have to control for number of factors that are really hard to control for in practice.

    I know that wasn’t central to your point, I’m just pointing out that what *justifies* someone’s being an expert may itself be rationalization (“truthization”).

    ***

    Regarding the label “troll”, it’s becoming like “terrorist”: yes, there’s a rigorous definition, but in practice, it will get thrown at whoever is out of favor.

    For my part, I’ve been called a “troll” for participation like this (posting under the handle “Person”), where all I do is point out problems in chains of logic. See also, how a well-respected thinker, Stephan_Kinsella, cannot even grasp the point I’m making when he responds, and no, it’s not a problem of poor phrasing.

  • http://greentheo.scroggles.com Theodore V.

    @Silas For my part, I’ve been called a “troll” for participation like this (posting under the handle “Person”), where all I do is point out problems in chains of logic. See also, how a well-respected thinker, Stephan_Kinsella, cannot even grasp the point I’m making when he responds, and no, it’s not a problem of poor phrasing.

    Likely if we were having face to face conversations with trolls we wouldn’t classify them as such. Hastily typed comments rarely express a fraction of the intended comment/idea.

    Sweeping Generalization. Experts are experts because they understand and are thoroughly familiar with the commonly accepted theories in their field. If they weren’t so familiar..they’d be called crackpots and potentially trolls. However history has a knack for turning Crackpots into geniuses and trail blazers.

    All that to say that given a good set of logical skills I agree with Sabbagh…what does an expert really have that I can’t legitimately disagree with even if I’m not nearly as expert as he may be? Data and interpretation are two vastly different things.

  • Constant

    1) There’s a “who will watch the watchers” worry to relying too much on experts and discounting what your own reasoning and senses tell you in favor of experts.

    2) Experts are experts because of knowledge hidden from non-experts. But when the experts lay out their case plainly, then this divide is bridged and non-experts become their equals. Expertise relies on hidden information, and it is not unreasonable to distrust hidden information.

    3) Experts love to venture outside of their area of expertise, and it’s not always obvious when they do it. This reason alone is reason to ask questions about the expert’s opinion. The opinion may in fact not be a genuine expert opinion.

    4) Experts disagree with each other frequently, and a non-expert who seems to be disputing the claims of one expert may, in fact, be relying on the claims of some other expert. In this case, the appearance of a dispute between an expert and a non-expert is an illusion – in fact what is going on is a dispute among experts, carried out by proxy through interested nonexperts, and both sides are expressing the judgment of expertise.

    5) There is a difference between different fields of studies. We know from experience that experts in physics are generally right. But it does not follow that experts in just any field are generally right. For example, an expert in astrology is very likely a believer in the validity of astrology, and so he is less likely to be right than a random lay person. So when contrasting experts with lay opinion, we need to consider the field in question.

  • michael vassar

    Rolf: I think that there are three meanings at issue, ‘intelligence’ in the casual sense, ‘reliability’, and ‘IQ’. Around the mean the three may be close enough in meaning that it would make sense for a random person of IQ 100 to confidently consider a random person of IQ 120 to be more ‘intelligent’ in the casual sense and also more reliable on random topics, but far from the mean it is not clear that this is the case. It’s hard to intuitively distinguish levels of ‘intelligence’ or reliability substantially greater than one’s own, but not at all hard to distinguish levels less than one’s own, and I think that many people may fairly asses Mr. Savant to be less ‘intelligent’ and ‘reliable’ than they are themselves or than the people from whom they have accepted their opinions are. Personally, I would say that I have taken on the opinions of the most reliable people I know in most topics, and that while I can fairly confidently judge Ed Witten to be more more ‘intelligent’ than myself or any of the people who I know I cannot make the same judgment of his ‘reliability’. Instead, it seems to me that several people I know, some of whom post here, are more ‘reliable’ than my estimate of the ‘reliability’ of a random person at the highest levels of IQ, such as Mrs Savant, or at the highest levels of ‘intelligence’, such as Newton. (honestly the first person to come to mind to me and many others as a possible “most intelligent person” but basically insane)

  • http://hanson.gmu.edu Robin Hanson

    Silas, it is reasonable to think an experienced TV producer’s estimates of TV show success are better than an average person, even if no study has rigorously proven it.

    Theodore, what makes you think your interpretation abilities are so good, that they usually make up for your lack of data?

    Constant, it is not reasonable to down-weight info simply because it is hidden. It is reasonable to distrust non-existent info. Suspecting they don’t know much is very different from expecting they know a lot, but not knowing exactly what they know.

    Michael, the point is that whatever indicators you have about reliability, you should defer to people who have stronger overall indicators, unless you have a good reason to think you are at an “enough” point where more doesn’t help.

  • Constant

    “Constant, it is not reasonable to down-weight info simply because it is hidden. It is reasonable to distrust non-existent info. Suspecting they don’t know much is very different from expecting they know a lot, but not knowing exactly what they know.”

    If it’s hidden from you, then by hypothesis you don’t know whether it is hidden or nonexistent, whether it is correct or erroneous, whether it is vast or meager, whether they know a lot or don’t know much, whether their reasoning is good or bad. And since you don’t know, then it is not unreasonable to distrust it.

  • http://hanson.gmu.edu Robin Hanson

    Constant, in poker you know your opponents know their own cards. The fact that you can see your cards and you cannot see their cards is not a good basis for assuming you know more about this round of poker.

  • Anon

    One man’s troll might be another man’s philosopher.

    Hanson shouldn’t be so hard on trolls or on people who discount expertise. In fact, Sabbagh’s comments may have touched a nerve with Hanson, who is a bit of a troll and discounter himself, especially in the field of medical care. And he should be applauded for that.

    Hanson is a troll vis a vis the health care establishment in that he calls into question the expertise of the medical profession in a provocative, argumentative way. When he claims that the U.S. could cut health-care spending in half and get the same quality of care, he is in effect calling some considerable fraction of the venerated medical profession incompetent at best, and perhaps fraudulent as well. A member of the medical community might rejoin that doctors have spent five to ten years acquiring their expertise and who is Robin Hanson, with not a day of medical school to his credit, to question how the experts practice medicine.

    Hanson might well be accused of several of the behaviors that he cites as troll-like, including:

    * a lack of buy-in to the list philosophy or values (of the medical establishment)

    * a mixture of friendly posts with a confrontational style of interaction (see his Cato debate)

    * the use of provocative language (“Cutting half of medical spending would seem to cost little in health . . . To their shame, health experts have not said this loudly and clearly enough.” See the Cato debate for more.)

    * a lack of in-depth understanding of the topic (where’s Hanson’s medical degree?)

    Of course some trolls are idiots, but not buying into a community’s values, mixing friendly conversation with a confrontational style and using provocative language sounds as much like a Socrates as it does like a bothersome troll.

  • michael vassar

    Robin: Part of my point is that plausibly many people really do substitute the opinions of others for their own when those others seem to display high reliability, but for the most part this simply pushes the problem to another level. Unless we can all agree on who is reliable, we still need to rely on our judgment at some level.

  • http://hanson.gmu.edu Robin Hanson

    Michael, we have been talking about using clues such as experience, detailed knowledge, and intelligence to estimate each person’s accuracy. Renaming accuracy as “reliability” doesn’t get us very far. And simply deciding that people you agree with are “reliable” and therefore worth listening to isn’t very useful either.

  • Caledonian

    I’m surprised at how many people let this point go without comment:

    a mixture of friendly posts with a confrontational style of interaction

    What a stupid criterion for trollishness. Haven’t you ever heard of the loyal opposition? Ignoring all of the other problems with the criterion, that alone is enough to sink it.

  • Constant

    “Constant, in poker you know your opponents know their own cards. The fact that you can see your cards and you cannot see their cards is not a good basis for assuming you know more about this round of poker.”

    Funny you should use poker as an analogy. In poker, since you don’t see the other guy’s cards, you don’t know whether he’s bluffing about his hand. So you reasonably distrust the signals he sends out.

  • komponisto

    Sabbagh: What experts have that I don’t are knowledge and experience in some specialized area…I now view the judgments of others, however distinguished or expert they are, as no more valid than my own .

    I suspect we have another case of underestimating the size of inferential distances. Perhaps Sabbagh thinks that an expert is just like a version of himself with a few more bits of data at hand; whereas in fact experts have much more than that: they have familiarity with long chains of reasoning that lead far away from our common everyday starting-points.

    Thus, Sabbagh might believe he has “just enough” expertise because he’s under the impression that more specialized training would only move a person at most one or two inferential steps away from where he already is (and thus would hardly be worth the trouble).

  • Jor

    One guy says it all: Phil Tetlock.

    Minimally informed people usually have as much expertise as “experts”, in domains with large uncertainties.

  • Caledonian

    Hmm, interesting.

    What the does ‘hedgehog’ metaphor have to say about Eliezer’s fixed ideas regarding Bayesian methods?

  • michael vassar

    And Robin, I have been pointing out, in response to Rolf’s point, that we don’t have any measures of domain general extreme “intelligence” that come even remotely close to corresponding to domain general extreme accuracy. As for experience and detailed knowledge, their relevance is also always still a judgment call.

  • http://entitledtoanopinion.wordpress.com/ TGGP

    Jor, laymen are not in fact just as good as experts as far as Tetlock shows. Bryan Caplan explained that in his review, which you can access through here.

    Michael Vassar, do you believe the psychometric theories of a single general intelligence g are wrong or only that our metrics for measuring it fall short?

  • michael vassar

    I don’t think that any theories even claim to know how g loaded accuracy of general beliefs is, but I think I made it clear from my posts that we don’t know how to measure g at the very top in any event. It’s obvious in a practical sense that, as I stated, Ed Witten is smarter than Maralyn Vos Savant regardless of what his IQ might be, and if you took a short list of standard candidates for “smartest person ever” based on IQ or genius accomplishments you would do worse in terms of finding accurate sets of beliefs than if you took, say, an average elite university natural scientist. Just look at the bios of the Mega society and the like, or of a list of important mathematicians (Newton, Godel…)

  • Silas

    komponisto: Perhaps Sabbagh thinks that an expert is just like a version of himself with a few more bits of data at hand; whereas in fact experts have much more than that: they have familiarity with long chains of reasoning that lead far away from our common everyday starting-points.

    Maybe *some* experts. I seriously doubt that most experts can explain all the logical steps in reasoning, starting from layman’s knowledge, that get to their current expertise (which, btw, is my criterion for establishing whether you understand something). In my admittedly limited experience, getting an expert to explain the context of their knowledge, and on what it is based, is like pulling teeth. They seem to operate in a sort of “Chinese room”, manipulating symbols to the satisfaction of others in the field, but not understanding them.

    Eliezer_Yudkowsky made a post a while ago explaining how it’s just not practical for an expert to explain to a layman all those steps. And I agree. But they should be *capable* of doing it, and that’s certainly the ideal, but I’ve just never seen it.

  • Nick Tarleton

    if you took a short list of standard candidates for “smartest person ever” based on IQ or genius accomplishments you would do worse in terms of finding accurate sets of beliefs than if you took, say, an average elite university natural scientist.

    I’m sure some (most?) of this is due to selection from different time periods. If you took the same candidates and compared them to a randomly selected elite academic from the last 2500 years, or alternatively compared the modern scientist to some candidate smartest people of the 20th century, the “smartest” groups would look a lot better, and the “smartest based on accomplishments” group might even win. Also, given the context of this discussion, you might want to exclude self-evidently crazy people like Godel, since you would exclude them if looking for the most generally reliable people.

  • Jor

    TGGP: I said minimally informed people, which means, people who read the new york times. I think if you also reference the recent discussion on this blog and Gelman’s blog about social science — you’ll realize Caplan’s larger point is wrong. Social Science experts often have a tendency to forecast well outside their knowledge, which makes them look foolish, and as Tetlock points out, no more accurate than a new york times reader. On forecasts people actually care about (i.e. the media actually asks experts about), experts often aren’t very good. If anything, I think this Tetlock’s point is one of the main reasons I favor Robin’s project, decision markets, than anything else.

  • Jor

    TGGP, let me also add, I think Caplan is right when he says experts can help frame questions properly, give choices, etc.. But their choices are often no better than flipping a coin in domains with larger uncertainties.

  • http://dl4.jottit.com/contact Richard Hollerith

    minimally informed people, which means, people who read the new york times

    If I might offer an alternative perspective: some say the Times is better at indoctrinating than informing.

  • http://hanson.gmu.edu Robin Hanson

    Let me put the challenge this way. Either there are indicators of who is more accurate or there are not. If there are no indicators, then everyone is equally likely to be accurate, and a simple average of everyone’s views would be the best we could do. If there are indicators of who is more accurate, then you should defer to the opinions of people who score better according to those indicators, in addition to feeling that those who score worse should defer to you.

  • http://entitledtoanopinion.wordpress.com/ TGGP

    I said minimally informed people, which means, people who read the new york times.
    If you read Caplan’s review, he points out that the “laymen” in Tetlock’s study were students who would have been far better informed than the general public.

    But their choices are often no better than flipping a coin in domains with larger uncertainties.
    Caplan also states in his review that Tetlock “loaded the dice”, as they say, in favor of the randomly-guessing “chimps” and against experts.