Dissing Track Records

Years ago I was being surprised to learn that patients usually can’t pick docs based on track records of previous patient outcomes. Because, people say, that would invade privacy and make bad incentives for docs picking patients. They suggest instead relying on personal impressions, wait times, “bedside” manner, and prestige of doc med school or hospital. (Yeah, those couldn’t possibly make bad incentives.) Few ever study if such cues correlate with patient outcomes, and we actively prevent the collection of patient satisfaction track records.

For lawyers, most trials are in the public record, so privacy shouldn’t be an obstacle to getting track records. So people pick lawyers based on track records, right? Actually no. People who ask are repeatedly told: no practically you can’t get lawyer track records, so just pick lawyers based on personal impressions or the prestige of their law firm or school. (Few study if those correlate with client outcomes.)

A new firm Premonition has been trying to change that:

Despite being public record, court data is surprisingly inaccessible in bulk, nor is there a unified system to access it, outside of the Federal Courts. Clerks of courts refused Premonition requests for case data. Resolved to go about it the hard way, Unwin … wrote a web crawler to mine courthouse web sites for the data, read it, then analyze it in a database. …

Many publications run “Top Lawyer” lists, people who are recognized by their peers as being “the best”. Premonition analyzed the win rates of these attorneys, it turned out most were average. The only way that they stood out was a disproportionate number of appealed and re-opened cases, i.e. they were good at dragging out litigation. They discovered that even the law firms themselves were poor at picking litigators. In a study of the United Kingdom Court of Appeals, it found a slight negative correlation of -0.1 between win rates and re-hiring rates, i.e. a barrister 20% better than their peers was actually 2% less likely to be re-hired! … Premonition was formed in March 2014 and expected to find a fertile market for their services amongst the big law firms. They found little appetite and much opposition. …

The system found an attorney with 22 straight wins before the judge – the next person down was 7. A bit of checking revealed the lawyer was actually a criminal defense specialist who operated out of a strip mall. … The firm claims such outliers are far from rare. Their web site … shows an example of an attorney with 32 straight wins before a judge in Orange County, Florida. (more)

As a society we supposedly coordinate in many ways to make medicine and law more effective, such as via funding med research, licensing professionals, and publishing legal precedents. Yet we don’t bother to coordinate to create track records for docs or lawyers, and in fact our public representatives tend to actively block such things. And strikingly: customers don’t much care. A politician who proposed to dump professional licensing would face outrage, and lose. A politician who proposed to post public track records would instead lose by being too boring.

On reflection, these examples are part of a larger pattern. For example, I’ve mentioned before that a media firm had a project to collect track records of media pundits, but then abandoned the project once it realized that this would reduce reader demand for pundits. Readers are instead told to pick pundits based on their wit, fame, and publication prestige. If readers really wanted pundit track records, some publication would offer them, but readers don’t much care.

Attempts to publish track records of school teachers based on students outcomes have produced mostly opposition. Parents are instead encouraged to rely on personal impressions and the prestige of where the person teaches or went to school. No one even considers doing this for college teachers, we at most just survey student satisfaction just after a class ends (and don’t even do that right).

Regarding student evaluations, we coordinate greatly to make standard widely accessible tests for deciding who to admit to schools. But we have almost no such measures of students when they leave school for work. Instead of showing employers a standard measure of what students have learned, we tell employers to rely on personal impressions and the prestige of the school from which the student came. Some have suggested making standard what-I-learned tests, but few are interested, including employers.

For researchers like myself, publications and job position are measures of endorsements by prestigious authorities. Citations are a better measure of the long term impact of research on intellectual progress, but citations get much less attention in evaluations of researchers. Academics don’t put their citation count on their vita (= resume), and when a reporter decides which researcher to call, or a department decides who to hire, they don’t look much at citations. (Yes, I look better by citations than by publications or jobs, and my prestige is based more on the later.)

Related is the phenomenon of people being more interested in others said to have the potential to achieve X, than in people who have actually achieved X. Related also is the phenomenon of firms being reluctant to use formulaic measures of employee performance that aren’t mediated mostly by subjective boss evaluations.

It seems to me that there are striking common patterns here, and I have in mind a common explanation for them. But I’ll wait to explain that in my next post. Till then, how do you explain these patterns? And what other data do we have on how we treat track records elsewhere?

Added 22Mar: Real estate sales are also technically in the public record, and yet it is hard for customers to collect comparable sales track records for real estate agents, and few seem to care enough to ask for them.

GD Star Rating
loading...
Tagged as: , , ,
Trackback URL:
  • http://ww.TobyUnwin.com Toby Unwin

    Great article, Robin.

    There are more news articles on Premonition here:

    http://premonition-analytics.com/news/

    • curious

      Found your post deeply interesting. Premonition seems like an excellent idea and may in fact do great work. However…

      I was interested in the company so I followed your link to their website and eventually got to the About Us – Team page. I googled the CEO to learn more – this is a great idea I wanted to see what else he’d done. First link that popped up was his Linkedin page. Way down at the bottom was his education: consisting of an MBA and BBA from Ashley University. I’d never heard of it – so googled – and turns out Ashley is a notorious diploma mill.

      From a Rhode Island channel 10 report: “I contacted Ashley University through its online chat, and quickly got a call back. Without any proof that I’d attended college, Ashley agreed to grant me a Ph.D. in political science. The cost was $648 and about 10 minutes of my time. The operator even offered to backdate my degree by as much as 20 years.”

      Again, Premonition may do great and real work – but this throws up all kinds of red flags about the entire organization.

      1) http://premonition-analytics.com/team-member/guy-kurlandski/

      2) https://www.linkedin.com/in/guykurlandski

      (scroll to the bottom for education)

      3) http://www.turnto10.com/story/25530806/i-team-higher-education-degrees-are-just-a-click-away

      • http://CommonSenseAtheism.com lukeprog

        I don’t see Ashley University on his LinkedIn; has it been removed? Can you post a screenshot?

      • http://CommonSenseAtheism.com lukeprog

        Google’s cache seems to think “Ashley University” was recently mentioned on the CEO’s LinkedIn page, even if it has since been removed:
        http://imgur.com/FLLQ6Ka

  • Frederic Bush

    Don’t all the mediocre performers have an incentive to keep their track records secret? Also, I think the best way to improve your trial grade would be to cherrypick your clients, which is problematic for unsympathetic defendants.

  • Andy McKenzie

    Worth pointing out the slight conflation in your first paragraph: completely agree that patient outcome data should be public, but patient satisfaction scores are *not* the same as this, and in some large studies satisfaction has been shown to be negatively correlated with both outcomes and positively correlated with expenditures: http://archinte.jamanetwork.com/article.aspx?articleid=1108766

    • http://overcomingbias.com RobinHanson

      Yes, satisfaction is not at all the same as health outcomes.

      • Faze

        University of Utah Health Care its now posting physician reviews and patient comments for all its docs, including a one-to-five-star rating system. Cleveland Clinic will soon go live with a similar system. The comments and reviews come from hospital-provided patient surveys, but the reviews and comments that are published are not cherry-picked. They appear alongside the docs’ bios in the online staff listings.

        http://healthcare.utah.edu/publicaffairs/news/archive/2012/12-10-2012_physician_reviews%20.php

  • Gwen

    “The system found an attorney with 22 straight wins before the judge – the next person down was 7. A bit of checking revealed the lawyer was actually a criminal defense specialist who operated out of a strip mall.”

    Sounds like someone I better call.

    • IMASBA

      With a win probability of 0.55 (0.50 would be pure chance) winning 22 cases in a row has a probability of about 1 in half a million. Sounds good doesn’t it? Well, just remember there are over a million living, licensed attorneys in the US alone…

      With a win probability of 0.60 the probability of a 22 cases winning streak is about one in 75 thousand (you’d expect at least a dozen such attorneys with winning streaks of 22 in the US). A 0.60 win probability is certainly better than random chance but it won’t deliver miracles and the attorney doesn’t have to be god’s gift to mankind to have that win probability.

      • Gwen

        It was a “Better Call Saul” reference.

      • IMASBA

        I know it was not a serious remark but you highlighted exactly the part that I wanted to comment on so I couldn’t resist.

      • http://overcomingbias.com RobinHanson

        But they don’t yet have data on anywhere near a million lawyers. And the 32 wins is one in a billion.

      • IMASBA

        And I’m sure the average active attorney has done many more than 22 cases in his or her career. Corruption and sample bias are also things you have to look out for (I’m sure it’s really difficult proving the guilt of shoplifters in a mall with all those security cameras, witnesses and very, very stupid shoplifters).

        P.S. with a win probability of 0.70 a 32 winning streak has a probability of 1 90 thousand (over a random string of 22 cases). The probability of win streaks rises exponentially with small increases in the win probability. Now every increase in win probability is good to have have but I’m pretty sure such small increments aren’t really what the customer thinks he’s paying for when he’s paying really big bucks for a “top” some-opaque-profession-practitioner.

      • http://juridicalcoherence.blogspot.com/ Stephen Diamond

        I’m sure it’s really difficult proving the guilt of shoplifters in a mall with all those security cameras, witnesses and very, very stupid shoplifters.

        This point seems so obvious that I don’t understand Robin’s high regard for the informativeness of track-record data. Professional track records are highly confounded.

        Perhaps the answer is Robin’s credo that data is better than no data. But this doesn’t hold for advertising! [Consider the proposal being entertained by the FDA to require labeling of the amount of sugar added to a product. More information! (And advocates don’t hesitate to argue the generic pro-data position.) But stupid because sugar is sugar.]

        The proposal would also have unfortunate incentive effects for client choice that Robin surprisingly neglects.

      • IMASBA

        Robin, take this opportunity. Watch Premonition’s database grow, carry out some critical statistical research et voila, you can do what you love doing most: be a contrarian and piss off people with biases.

        You know you want to…

    • RobertB

      Having an outlier win streak before a particular judge (as opposed to winning cases generally) kinda suggests corruption more than anything else.

  • IMASBA

    This behavior ultimately isn’t that irrational: most careers are too short to base good statistics on and even when that’s not the case the information is incredibly noisy with survivorship bias, luck with the kind of cases and other forms of luck playing a large role. Also no lawyer, teacher or doctor on the planet has an IQ of 300 (and even if one did, that only affects the small part of the game that isn’t based on luck): gains from getting a better one are limited and diminishing returns kick in quickly. In light of this it’s actually quite rational to go with someone you like: at worst it makes your cooperation more pleasant, at best it actually improves the outcome a bit by preventing fights and (unconscious) demotivation. Miracle workers do not exist: it’s easy to screw up, it’s very hard to overcome randomness and produce something with some consistency, it’s impossible to produce a miracle except when the gods of randomness decides to give you one.

    Now there are is a form of irrationality to be found here: people typically don’t know about the above: they do believe in miracle workers (and of course the higher the fee, the greater or more frequent the miracles). Perhaps one part of our mind involved here is more BS-free than the other (our intuition, shaped by surviving millions of years of evolution, that guides the choice being honest about miracle workers not existing, while our conscious excuse for the choice is a belief that miracle workers are nice), or perhaps it’s just a coincidental example of two irrationalities making a rationality (believe in miracle workers and the believe that miracle workers are nice people leads to people randomly selecting workers they can get along with, a result that is somewhat rational).

    • Grant

      If things are as random as you suggest, the numbers will reflect this.

      • IMASBA

        They already do for the stock market (to a high degree: skill can sometimes make a difference, as it does in poker, but it’s very limited) and I’m quite convinced Premonition’s data will ultimately show the same thing for attorneys and teachers.

    • Ronfar

      > they do believe in miracle workers (and of course the higher the fee, the greater or more frequent the miracles)

      O.J. Simpson certainly got his miracle… There really is a big difference between what a millions-of-dollars legal defense team can do and a public defender can do.

      • IMASBA

        A popular celebrity who paid a team of the same number of “normal” lawyers some fee that they can live off of comfortably would have a near equal probability of getting their “miracle”, though they’d miss the self-fulfilling prophecy effects that a “top” reputation can bring. That last effect as well as the celebrity effect are excecarbated by judicial systems that rely heavily on laymen juries.

      • http://juridicalcoherence.blogspot.com/ Stephen Diamond

        A popular celebrity who paid a team of the same number of “normal” lawyers some fee that they can live off of comfortably would have a near equal probability of getting their “miracle”, though they’d miss the self-fulfilling prophecy effects that a “top” reputation can bring.

        I think that’s entirely false. (What are your reasons?) Would normal lawyers have come up with “if it doesn’t fit, you can’t convict”?

        “Normal” lawyers are creatures of routine.

        Let me ask you this. There are (wouldn’t you agree?) towering scientific geniuses. Why shouldn’t there be towering business geniuses and legal geniuses?

      • IMASBA

        “Let me ask you this. There are (wouldn’t you agree?) towering scientific geniuses. Why shouldn’t there be towering business geniuses and legal geniuses?”

        There have been very few “towering scientific geniuses” in the history of the world. Einstein may have been one (he was instrumental in multiple fields and had multiple big ideas, unlike most other famous scientists who were one-hit wonders), but overall competition is very though (even average scientists are very smart) and almost everything depends on being in the right field at the right time.

        Are there Einsteins of business and law? Sure there have to be some, but these fields are even messier than science (Einstein could basically work out special relativity behind his desk at home, the math and physics either fit or they do not, if the world’s best business or legal mind would be working as a patent office clerk there’s no way he’d ever got to put that mind into practice). I certainly do not believe all those thousands of “top” businessmen and lawyers in the world have Einstein-caliber minds, but the whole point is that even if they are that smart and creative that really doesn’t matter very much with all the random noise and viral effects in those fields.

        I don’t see what people find so genius about the Chewbacca defense. Confusion through overwhelming with nonsensical arguments has been known as a tactic since at least the ancient Greeks. Btw, has everyone forgotten that OJ Simpson lost a civil case about the murders and is currently serving a long sentence for other crimes? He only got a miracle one out of those three times even though he is a celebrity and each time had multiple experienced attorneys (say win probability of 0.65) on his payroll.

  • http://praxtime.com/ Nathan Taylor (praxtime)

    When I choose a doctor or a school for my kids, my instinctive heuristic is recommendations from friends. The subtlety here is that recommendations are really about coalition politics (are you on my team), and not objective measured results. Hence we should not be surprised to find that people are far more interested in promoting their school as an alumni than measuring in an objective fashion whether that school is good or bad. I have kids in grade school and middle school, and when people I know discuss whether to go to private school, the most persuasive case is made by explaining which schools make you feel good or bad or which parents also agree that these schools are good or bad. It’s not about the numbers (test scores). It’s about which team (school) will win. School sports have deep emotional pull for precisely this reason. Mock battles to show which team is superior.

    You can put an evolutionary gloss on this, proposing why humans are so finely attuned to coalition politics. Which makes sense to me, even though I think that is speculative for now. But however much this is cultural versus genetic, team allegiance appears to be a universal heuristic for decision making. What’s far more difficult is to suppress this instinct to look at objective numbers of the results you (should hopefully) care more about.

  • Jess Riedel

    I have some criticism of individual points. I know you’re interested in an overarching theory and would likely dismiss these as minor complications.

    > … patients usually can’t pick docs based on track records of previous patient outcomes. Because, people say, that would invade privacy and make bad incentives for docs picking patients. They suggest instead relying on personal impressions, wait times, “bedside” manner, and prestige of doc med school or hospital. (Yeah, those couldn’t possibly make bad incentives.)

    Three points:

    (1) The docs I know don’t recommend looking at any of those four things. The recommend asking other doctors. This is true *even when they need treatment for themselves*. However, they *do* (at least claim to) take outcomes into account when they are personally familiar with the sort of patients the doctor has treated.

    (2) Depending on the operation, the noise from sample bias is very often much larger than the small signal from doctor quality. Your argument would do better if you concentrated on the clear examples where sample bias is low.

    (3) Record tracking would have at least two *overall* effects on the market:
    (a) bad incentives for docs to avoid difficult patients
    (b) improved average patient outcomes as bad docs are more thoroughly weeded out
    The first order question is whether (a) > (b), and you haven’t even considered the rough magnitude. The possibility that individual patients do better using track records (which would be in tension with point (1)) isn’t a great argument for record tracking from a group perspective. Negative sum game, etc.

    >Few ever study if such cues correlate with patient outcomes, and we actively prevent the collection of patient satisfaction track records.

    The fact that this isn’t better studied is surprising and bad, and does cast some doubt on arguments like mine above.

    > They discovered that even the law firms themselves were poor at picking litigators. In a study of the United Kingdom Court of Appeals, it found a slight negative correlation of -0.1 between win rates and re-hiring rates, i.e. a barrister 20% better than their peers was actually 2% less likely to be re-hired!

    Isn’t it a lot more likely that their metric was bad (i.e., sample bias makes track record a bad indicator of lawyer quality), than that the law firms with money on the line were all idiots?

    • http://overcomingbias.com RobinHanson

      I doubt anyone studies if “ask other docs” correlates with outcome; why don’t they care? Overall system effects of track records can’t much explain why individuals aren’t interested; they personally don’t pay for such system costs. I’ll bet you that the barrister study isn’t bad.

      • Jess Riedel

        > I doubt anyone studies if “ask other docs” correlates with outcome; why don’t they care?

        Agreed that this needs explanation, although note that the barrister study sort of qualified. (The “Top Lawyers” list is based on peer recommendations.) I look forward to the Washingtonian’s “Top Doctors” list being similarly studied.

        > Overall system effects of track records can’t much explain why individuals aren’t interested; they personally don’t pay for such system costs.

        Agreed. Here’s one explanation: It’s plausible to me that tracking medical records only became feasible in past few decades, and that usefully large enough data appeared only in last 10 or 15 years. So choosing based off of reputation only recently become a marginally non-optimal strategy, of which people are rationally ignorant.

        With regard to your overarching question, this generalizes to: “sometimes (biased) reputations systems are better and sometimes (imperfect) metrics are better; we should expect metric systems to become relatively better as data sets improve”.

        > I’ll bet you that the barrister study isn’t bad.

        I agree that win rates are useful data, and I think the barrister study will find use. However, unless win rates are conditioned on key parameters (type of case, judge, etc.), I claim they are significantly less useful than reputation due to things like sample bias. The quote I criticized was about unconditioned win rates, and I extend that criticism to the related statements in the article.

        I am certainly happy to bet about that. Not sure what your terms would be since it is unlikely we’ll get an RCT of reputation-picked vs. unconditioned-win-rate-picked lawyers.

  • Grant

    Law and medicine seem similar in other regards. They’ve both protected by government-industry collusion which even have similar initials, the ABA and AMA. Entrance into both professions is restricted by schools and licensing, and the public seems largely happy with this (maybe because they don’t realize how much the supply of doctors is being restricted*).

    Robin, I’m suggesting your two examples, medicine and law, don’t suggest that consumers diss track records in general. It could be the lack of doctor and lawyer track records is a symptom of the amount of protectionism workers in those industries enjoy. Consider the recent revolution in online seller track records, largely driven by Amazon and eBay.

    Or, maybe the high status of doctors and lawyers affords them both political power and the ability to be jerks to customers (long wait times, refusal to publish track records, etc.). This seems less likely to me though, since I can’t think of high-status but politically disconnected jobs where track records aren’t used (athletes are a prime example).

    * http://mjperry.blogspot.com/2010/04/medical-school-grads-have-been-flat.html#links

    • IMASBA

      There’s probably truth in this (some European governments even literally restrict the annual number of accepted applicants at medical schools, though these governments also pay for these schools and the medical system, so it’s more complicated than just cronyism), though it can’t be the whole story, otherwise there’d be popular protests and such…

  • lemmycaution

    People do select mutual fund managers with the tactics that you suggest. There is a huge random component in the outcomes so it isn’t really a good model for other professions (or even for mutual fund managers).

    • Michael Vassar

      Mostly they don’t though, even in mutual funds, or there couldn’t possibly be so many mutual funds, as no-one would invest in any except the few best, since that’s so objectively easy to analyze.

  • BJ Terry

    Related to your reference on tracking pundits, there is a startup that tries (tried?) to track pundit predictions (http://pundittracker.com/). It sort of supports the opinion of the media firm, however. Since I last looked they have completely changed their site and now it only seems to track the predictions of users rather than actual pundits, which are now only tracked on their blog. This is possibly from not getting enough interest in pundit predictions so trying to sprinkle more magic social dust on their business model.

  • Robert Koslover

    Most people are only marginally-competent in their occupations, and many of them know it. It makes sense that most people shy away from scrutiny, lest their less-than-stellar performance and/or abilities be revealed. Let A = the advantages (speculative) that we might all reap by scrutinizing others. Let B = the advantages (more genuine) that we individually reap by not being scrutinizing too closely. I suggest that we are more than willing to give up A in exchange for B. (And I also think this principle applies more widely than your examples.)

    • Robert Koslover

      Typo: “that we individually reap by not being scrutinizing” should have been “”that we individually reap by not being scrutinized”

    • http://juridicalcoherence.blogspot.com/ Stephen Diamond

      It makes sense that most people shy away from scrutiny, lest their less-than-stellar performance and/or abilities be revealed.

      There are many things we dislike yet impose on others. You’re crediting humans with a great deal of empathy to say I won’t scrutinize you because I don’t want to be scrutinized. (But perhaps homo hypocritus says just this?)

  • Phil

    As Robert Koslover says, nobody wants to be rated relative to their peers. That way lies loss of status. It’s not even zero-sum — it’s *negative* sum. The below-average performers lose more than the above-average performers gain.

    Before evaluations, all teachers were high status. Now some are still high, but most are lower, and only a select few are very high.

    Why do CONSUMERS not want this data? They’re scared of making bad decisions. They’re scared that if they’re not careful, they might wind up following someone of low status, and be thought of as low status themselves. They’re scared they might be the ones stuck with the crappy doctors.

    So everyone has an incentive to pretend that everyone is of equal quality.

    • http://overcomingbias.com RobinHanson

      But the indicators that we do, like degrees, use instead of looking at outcomes, have all these same problems.

      • Phil

        I don’t know if degrees have those same problems. Do they? For the people I hold in high esteem — certain bloggers, authors, etc., — I have no idea what degrees they have or where those degrees came from. (Including yourself.)

        Also, with degrees, once you achieve them, your status is assured. You can’t ratchet down. If you become a worse doctor over time, nobody knows — you’re still a doctor from Harvard. You can relax and not have to work so hard to compete with other doctors in real time. The competition was getting into Harvard in the first place.

  • The Do-Operator

    Suppressing this information may be the price society pays in order to incentivize the best candidates to go into these professions.

    Suppose your performance as a physician is a hidden variable that is unknown to you until after you qualify as a doctor. Moreover, suppose nobody will want to be your patient if it turns out that your performance is on the lower end relative to other doctors

    Training as a doctor takes 4 years of college, 4 years of medical school and at least 3 years of residency. This is a serious financial and personal investment. Not being able to practice medicine after making such an investment is a financial catastrophe and a deep personal tragedy. If this investment is expected to become worthless to half of medical students because they are in the lower half of their class, it will seriously reduce the incentives to become a doctor. Also note that we are talking about relative measures of quality, there will always be a lower half of the class.

    This is particularly problematic if the hidden variable for your performance as a doctor increases steeply in the lower range of the population, but is flat at the top. We may have found a way to select doctors who are only from the flat end of that curve, but if we increase the risk of the investment massively, the cutoff-point may be pushed back into the steep segment of the curve

    (This is obviously a thought experiment, I certainly do not believe we have found a way to select doctors only from the flat end of the curve)

    • http://overcomingbias.com RobinHanson

      This argument applies to ANY info we let patients see about docs. So why don’t we hide the info we now reveal?

      • arch1

        Because *that* level of info is just right: Enough to placate consumers without being sufficiently informative to cause big differences in expected earnings for most docs (and OBTW likely reduce patient death rates, but since when has *that* been of paramount importance:-).

      • http://overcomingbias.com RobinHanson

        The school info we release about lawyers does cause huge differences in earnings.

      • The Do-Operator

        I’m not sure exactly what you mean by “school info”, but if you mean which school they attended, then this is something that is known to applicants at the time they decide whether to make the investment. It affects the expected return, but not the risk.

  • dvasya

    In academia many professors maintain lists of alumni with current positions of their former students and postdocs. For someone choosing an advisor with an academic career in mind this could be a helpful “track record”.

  • Sam Dangremond

    Maybe “outcomes” aren’t the value-added customers seek from certain service providers?

  • Tom

    There is a potential perverse incentive here for lawyers to pick easy cases or for doctors to avoid troublesome patients. It’s easy to game the system by cherry-picking cases to boost stats.

    For example, federal prosecutors have conviction rates in the high 90s. You won’t find a law firm in America with those kinds of results. Does that imply that the Attorney General’s office attracts or produces great attorneys? Or is it a sign that something else is going on?

    I think that there may be some intuitive understanding of this kind of phenomenon that prevents it from being commonplace. We want doctors who can save people at the brink of death and lawyers who can handle tough cases. No one gets good at those without amassing a ton of failures along the way.

  • curious

    Found your post deeply interesting. Premonition seems like an excellent idea and may in fact do great work. However…

    I was interested in the company so I followed your link to their website and eventually got to the About Us – Team page. I googled the CEO to learn more – this is a great idea I wanted to see what else he’d done. First link that popped up was his Linkedin page. Way down at the bottom was his education: consisting of an MBA and BBA from Ashley University. I’d never heard of it – so googled – and turns out Ashley is a notorious diploma mill.

    From a Rhode Island channel 10 report: “I contacted Ashley University through its online chat, and quickly got a call back. Without any proof that I’d attended college, Ashley agreed to grant me a Ph.D. in political science. The cost was $648 and about 10 minutes of my time. The operator even offered to backdate my degree by as much as 20 years.”

    Again, Premonition may do great and real work – but this throws up all kinds of red flags about the entire organization.

    1) http://premonition-analytics.c

    2) https://www.linkedin.com/in/gu

    (scroll to the bottom for education)

    3) http://www.turnto10.com/story/

    • Robert Koslover

      Hmm. Your links didn’t work. But I went to the Premonition website, also tried LinkedIn, scrolled down, did a search for “Ash” etc., and can’t find anything like what you said. Maybe it has been changed? I separately found your info about Ashley Univ on a different website, but with no connection to the Premonition CEO. Anyway, could you please post again, with specific and working links? Thanks.

  • lump1

    My two thoughts on the subject:

    Lots of easily accessible performance info leads to more people making decisions based on that info. And when that happens, that raises the incentive to game the info instead of just doing your job well. In the case of lawyers, they would obviously steer away from cases where winning is a longshot, and such an incentive would be a public evil. Surgeons would refuse high-risk patients. Etc. We don’t want to give professionals a reason to game their stats, or to adjust their behavior for the mere sake of upping their score. One way to do this is simply to not keep score. I’m not saying that this is a sound rationale, but it’s not senseless. (Edit: Tom made a similar point 20 minutes before, I just didn’t see it.)

    My second thought is that we may not be clamoring for performance info, but when we have it, we definitely use it. For example, consider TripAdvisor. Before it came along, travelers had their own cocktail of heuristics for making good hotel and restaurant decisions. We were having perfectly wonderful holidays and even enjoyed not knowing exactly what to expect when we arrived. Nowadays, nobody books anything before checking TripAdvisor. Also, every undergraduate I know reads professor reviews before registering for a class. I didn’t have that option, so I relied on an anecdotal reputation system that I didn’t see as being broken or inadequate. It’s possible that we were making worse travel, professor and dating decisions before the era of data saturation, but it seemed to us like we were getting what we wanted, so there wasn’t much demand for more public information. (Maybe also, we wanted our special discoveries to stay secret, so that they wouldn’t become profaned.)

  • http://juridicalcoherence.blogspot.com/ Stephen Diamond

    While abstaining collecting track record information might be rationally justified, this still doesn’t explain our avoidance of providing this specific information. So, I still get to provide a theory.

    Why in a success-oriented culture are we uninterested in collecting data about success? Viewed this way, the answer would seem to be the same as the answer to another question: why don’t we collect (or present) income data?, even as accrued professionally? Both to the layman and the economist, market value measures social contribution. Wouldn’t lawyers’ income be a better indicator of their skills than their success records?

    What do the two have in common? They both express a forager’s hatred of arrogance and boasting. Then why can we list universities attended? There’s some camouflage: universities don’t come with explicit labeling of elite status.

  • K

    There are people who really do try to measure real teacher effectiveness at the college level. However, student evaluations are not the right way to do it, in part because students are terrible at really understanding what helps them learn (that may be in your other post, I didn’t click through). What we’re finding is that if a class is really hard with high expectations (and hence lower student evaluations than an “easy” professor) that those students have better outcomes later (in future classes). However, administration just wants to use student evals rather than the more intensive complex evaluations of teachers. Because it’s easier. This of course does not drive anywhere near the right behavior in either students or professors (unless we’re willing to just ignore what the students are likely to say, which is fine if you’re tenured, and less fine if you’re adjunct or pre-tenure).

    • K

      (and of course, none of that’s public, which is also your point)

  • arch1

    Perhaps others can confirm/deny/provide cites: I seem to recall that for surgical procedures, the number of such procedures performed by a surgeon has a sizeable positive correlation w/ goodness of outcome.
    Such info would seem to be fairly easy to get a ballpark handle on.

  • EdReal

    “Parents are instead encouraged to rely on personal impressions and the prestige of where the person teaches or went to school. ”
    No, not at the K-12 level. Quite the contrary. The entire reform movement is spent working desperately to convince parents to care about metrics like test scores and academic outcomes. It doesn’t work.
    Parents know exactly what they want from a school, and that’s a desirable peer group for their kids–as good as they can pay for or convince someone to give them (in the form of urban charters).
    Parents know intuitively that most teachers will be able to deliver an education if they have a class of kids willing and ready to learn. Teacher metrics only occasionally capture actual teacher quality, and then only when compared to other teachers with similar class compositions.
    I suspect this is true for doctors and lawyers as well, that their success rate depends on their clients.
    Moreover, if we really could capture doctor and lawyer quality and make that information public, they’d charge more and only the very richest could afford them. The average joe is better off hoping quality is somewhat randomly distributed.

    • http://overcomingbias.com RobinHanson

      The prestige of where the person teaches captures the peer group quality.

      • EdReal

        But that’s the point. The “teacher quality” isn’t even in the equation. They aren’t saying “this teacher must be good because she’s teaching at a high quality school”, but rather “the students at this school all come from highly educated families, therefore it doesn’t matter if the teacher is objectively high quality”.
        GOing further, most parents don’t agree that there is such a thing as an objectively high quality teacher–and they’re probably right.
        So 1) parents aren’t encouraged to rely on personal impressions and prestige. They are encouraged (by ed reformers) to use hard data. Parents reject this. and 2) parents understand full well that teaching quality is not particularly related to student outcomes, which is why they do it. They understand that the metrics you want them to use are pointless.

      • http://juridicalcoherence.blogspot.com/ Stephen Diamond

        There’s some ancient research showing that the intellectual level of fellow students is the strongest predictor of gains.

    • IMASBA

      Yes, parents care mainly about the (status of) the peer group and the ideology of a school. Raw performance barely enters the equation, except when choosing a school within a certain system (but usually there aren’t too many of those in the same neighborhood). This even happens when the parents know they’re actually hurting the chances of the child on the future job market.

  • Will

    Premonition has made an obvious mistake (there are other companies in the law big data space who avoid it)- they modeled outcome without controlling for difficulty. The reason that top lawyers only appear average is that they routinely get harder, higher stakes cases (also why the litigation lasts longer).

    The fact that podunk lawyer in the strip mall can rack up 22 straight wins is that he is representing minor crimes or people who just need a letter to their landlord or whatever.

    • IMASBA

      The list of things that should be controlled for (and how they should be controlled for in practice: how do you measure the “difficulty” of a case and is the required data even available?) is a never-ending discussion that you’re always going to run into when you try to distill performance into one figure. The big question Hanson has exposed here is why we’re giving some professions such as attorneys and teachers much more leeway than others when it comes to measuring performance (society isn’t exactly squeemish about judging people in other professions based on a single performance figure).

      • http://juridicalcoherence.blogspot.com/ Stephen Diamond

        society isn’t exactly squeamish about judging people in other professions based on a single performance figure

        Which other professions?

        This is a claim that would rebut Robin’s analysis. I don’t think it’s correct: the most definitive performance stat, income, is seldom flaunted.

      • IMASBA

        Sports, the business world, construction.

      • http://juridicalcoherence.blogspot.com/ Stephen Diamond

        Robin just added real estate.

        You’re usually more loquacious…

      • IMASBA

        The real estate thing is quite alien to me. Here in Europe we simply don’t look for the “best” real estate agent. We simply see the agent as a menial middle man, with the house seling itself, perhaps this is because it’s easier to look up government estimates of a house’s value, government requirements to have an independent taxation before a sale and the selling prices of local houses being quite easily accessible public information.

    • mel belli

      All wrong. In reality big firm, prestige lawyers often work very easy cases, and when they get a tough one, it’s still an easy one because the sophisticated client doesn’t expect much, hoping only to delay payment or for some kind of hail-mary like a credulous judge. Examples are easy to supply, just ask.

  • Anand Vemuri

    With the exception of pundits, most of these professions are rival in consumption. We have no trouble with relative ranking when it comes to nonrival consumption of performance (composers/quarterbacks/artists). We do debate who is the best of all time and eagerly consume rankings, however imperfect.

    With rival consumption, it’s a mathematical fact that if supply is proportionate to demand half the people will be served by a below median doctor/lawyer (assuming capacity is the same across quality). It isn’t comfortable to most people to know that their doctor is mediocre, or their kids are being taught by mediocre teachers. Yes, degrees and bedside manner etc also create relative rankings, but in aggregate they provide a rationale to why your choice is acceptable. [good school OR good bedside manner OR good hospital OR …] probably covers pretty much whatever choice you made.

    Pundits are an interesting case in that they’re nonrival. I’d speculate the mechanism there is slightly different. Although there is no economic cost to switching to a better pundit (unlike a better doctor), there is a high psychological cost since your identity is invested in your favourite pundit’s opinions.

    • http://overcomingbias.com RobinHanson

      Yes school degrees “provide a rationale” but then so would outcome track records. This criteria doesn’t distinguish between them.

      • Anand Vemuri

        My point was relative rankings cause cognitive dissonance. With other vaguer factors, you’ll find some criteria or the other by which you’ve made a good choice. Objective data on outcomes makes you confront ugly realities.

        Typical rationalizations go like this

        “He didn’t go to a great school, but he really cares about me”
        “He’s really rude, but he went to Harvard”

        These allow patients to rationalize their choices since there is sufficient vagueness in cause and outcome.

        Relative rankings remove that fuzz factor and cause cognitive dissonance when you want to believe you’re in good hands and/or have made a good choice. “He’s in the bottom quartile of doctors in terms of outcomes, but he went to Harvard” doesn’t have the same ring to it.

      • http://overcomingbias.com RobinHanson

        I talked about outcomes not rankings. You can rank any of the kinds of cues that people use, not just outcomes.

  • Pingback: Quotes & Linnks #54 | Seeing Beyond the Absurd

  • Foo McBarson

    It’s not hard to explain why top doctors, lawyers, etc. resist evaluation according to new metrics; by definition, the “top” lawyers/doctors are those that rank highly according to existing criteria, so the institution of new criteria constitutes a threat to their status. The employers of doctors/lawyers are themselves top doctors/lawyers, so it’s not especially surprising that there’s no push from employers. You point out that customers don’t seem interested in track records, but I think your observation is premature. It’s normal for a startup to have trouble recruiting customers even when its product is superior. Rule of thumb is you want to be 10x better than what already exists in the marketplace. Anything else means an uphill slog of marketing/“starting a movement” in the sense of https://web.archive.org/web/20141009211917/http://www.gabrielweinberg.com/blog/2010/02/are-you-building-an-empire-sparking-a-powder-keg-or-starting-a-movement.html Note that there are examples of this sort of performance based evaluation taking over, but it doesn’t happen automatically; Moneyball is a good example (and there are fairly obvious reasons we should expect it to be slower in other fields, I think). So, short answer: humans are impressed by whichever attributes are currently statusful, and we are also unduly influenced by the opinions of statusful people, which results in the currently statusful attributes being self-reinforcing. This explains why e.g. we’re more willing to grade high school teachers according to strict quantitative criteria, since their profession is a less statusful one than that of the college professor.

  • Phil H

    For several of the major professions, rating individuals by performance would be antithetical to the nature of the system.
    Law is a very clear example: justice is supposed to be fair for everyone. The goal of the system is that you don’t get an outcome based on which lawyer you have, you get the outcome you deserve based on the facts and the law. To introduce a lawyer ranking system over the top of that would be wildly inappropriate – it would contribute to turning the justice system into nothing more than a sophisticated game.
    In medicine and education the situation is similar. The goal is not that there should be some good teachers, available to the rich or the lucky. The goal of the system is that every child should have access to decent teachers. To *systematically* rank and sort teachers by quality would be a grotesque admission that some kids just ain’t gonna get the best. Medicine the same: I don’t want to have to sort through the rankings. I want a competent, nearby, affordable doctor.
    I don’t dispute that there is a significant element of class snobbery in opposition to rankings; but there are also very good reasons for not having them in many fields.

  • Pingback: Friday Links « Healthcare Economist

  • Pingback: A week of links | EVOLVING ECONOMICS

  • dmytryl

    Well, for one thing, if you rank lawyers based on their win percentages your ‘top lawyers’ will be the lawyers who found a way to end up with the easiest cases, and nothing more than that.

    An idiotic idea threatens everyone’s existing status, but that’s not why the idea is not adopted; it is not adopted simply because it is idiotic.

  • Pingback: Why are track records unpopular? | Meteuphoric

  • Pingback: Overcoming Bias : Sycophantry Masquerading As Bargains

  • Pingback: Overcoming Bias : Sycophantry Masquerading As Bargains