(No I’m not going to say more about the book now.)
(No I’m not going to say more about the book now.)
Years ago I was being surprised to learn that patients usually can’t pick docs based on track records of previous patient outcomes. Because, people say, that would invade privacy and make bad incentives for docs picking patients. They suggest instead relying on personal impressions, wait times, “bedside” manner, and prestige of doc med school or hospital. (Yeah, those couldn’t possibly make bad incentives.) Few ever study if such cues correlate with patient outcomes, and we actively prevent the collection of patient satisfaction track records.
For lawyers, most trials are in the public record, so privacy shouldn’t be an obstacle to getting track records. So people pick lawyers based on track records, right? Actually no. People who ask are repeatedly told: no practically you can’t get lawyer track records, so just pick lawyers based on personal impressions or the prestige of their law firm or school. (Few study if those correlate with client outcomes.)
A new firm Premonition has been trying to change that:
Despite being public record, court data is surprisingly inaccessible in bulk, nor is there a unified system to access it, outside of the Federal Courts. Clerks of courts refused Premonition requests for case data. Resolved to go about it the hard way, Unwin … wrote a web crawler to mine courthouse web sites for the data, read it, then analyze it in a database. …
Many publications run “Top Lawyer” lists, people who are recognized by their peers as being “the best”. Premonition analyzed the win rates of these attorneys, it turned out most were average. The only way that they stood out was a disproportionate number of appealed and re-opened cases, i.e. they were good at dragging out litigation. They discovered that even the law firms themselves were poor at picking litigators. In a study of the United Kingdom Court of Appeals, it found a slight negative correlation of -0.1 between win rates and re-hiring rates, i.e. a barrister 20% better than their peers was actually 2% less likely to be re-hired! … Premonition was formed in March 2014 and expected to find a fertile market for their services amongst the big law firms. They found little appetite and much opposition. …
The system found an attorney with 22 straight wins before the judge – the next person down was 7. A bit of checking revealed the lawyer was actually a criminal defense specialist who operated out of a strip mall. … The firm claims such outliers are far from rare. Their web site … shows an example of an attorney with 32 straight wins before a judge in Orange County, Florida. (more)
As a society we supposedly coordinate in many ways to make medicine and law more effective, such as via funding med research, licensing professionals, and publishing legal precedents. Yet we don’t bother to coordinate to create track records for docs or lawyers, and in fact our public representatives tend to actively block such things. And strikingly: customers don’t much care. A politician who proposed to dump professional licensing would face outrage, and lose. A politician who proposed to post public track records would instead lose by being too boring.
On reflection, these examples are part of a larger pattern. For example, I’ve mentioned before that a media firm had a project to collect track records of media pundits, but then abandoned the project once it realized that this would reduce reader demand for pundits. Readers are instead told to pick pundits based on their wit, fame, and publication prestige. If readers really wanted pundit track records, some publication would offer them, but readers don’t much care.
Attempts to publish track records of school teachers based on students outcomes have produced mostly opposition. Parents are instead encouraged to rely on personal impressions and the prestige of where the person teaches or went to school. No one even considers doing this for college teachers, we at most just survey student satisfaction just after a class ends (and don’t even do that right).
Regarding student evaluations, we coordinate greatly to make standard widely accessible tests for deciding who to admit to schools. But we have almost no such measures of students when they leave school for work. Instead of showing employers a standard measure of what students have learned, we tell employers to rely on personal impressions and the prestige of the school from which the student came. Some have suggested making standard what-I-learned tests, but few are interested, including employers.
For researchers like myself, publications and job position are measures of endorsements by prestigious authorities. Citations are a better measure of the long term impact of research on intellectual progress, but citations get much less attention in evaluations of researchers. Academics don’t put their citation count on their vita (= resume), and when a reporter decides which researcher to call, or a department decides who to hire, they don’t look much at citations. (Yes, I look better by citations than by publications or jobs, and my prestige is based more on the later.)
Related is the phenomenon of people being more interested in others said to have the potential to achieve X, than in people who have actually achieved X. Related also is the phenomenon of firms being reluctant to use formulaic measures of employee performance that aren’t mediated mostly by subjective boss evaluations.
It seems to me that there are striking common patterns here, and I have in mind a common explanation for them. But I’ll wait to explain that in my next post. Till then, how do you explain these patterns? And what other data do we have on how we treat track records elsewhere?
Added 22Mar: Real estate sales are also technically in the public record, and yet it is hard for customers to collect comparable sales track records for real estate agents, and few seem to care enough to ask for them.
I should be on tonight’s (9pm EST) episode of Stossel, on Fox Business TV, talking about biases.
Added 8Dec: I was wrong; the show should air Thursday Dec. 11
Added 28 Dec: Here is a video of the episode
Imagine that you wanted to write a popular book on the anxiety de jour, and that this anxiety happened to be increased moral depravity. Well there’d be an easy time-tested recipe to follow.
First, give some plausibility arguments for why moral depravity is a big deal. Since everyone already thinks so, weak arguments would be fine. Second, give lots of concrete examples of people and orgs affected by moral depravity, examples readers can relate to. Especially examples about high status and new things – people love to read about those. Third, mention important recent worrying trends backed up by serious research, and vaguely suggest that these trends are caused by increased moral depravity. No need for concrete arguments, you just need to show you are a serious person tracking serious trends. Finally, recommend a bunch of policies to deal with moral depravity, policies many of your readers already support, and that you would have supported even if every one of those recent trends were opposite.
Most important: have your book come out just as talk about moral depravity was peaking, and be an author with a lot of status in reader eyes. Your readers would mainly just want a book they could point to as they argue the topic, so they’d mainly just want an easy read without subtle arguments that they could fail to understand.
This is the recipe that Erik Brynjolfsson and Andrew McAfee follow in their new book The Second Machine Age. They are high status authors, and their book arrives just as computer anxiety is peaking. First, they suggest that computers will cause an economic revolution as big as the industrial revolution, which they say was caused by the steam engine. Second, they review lots of fashionable new computer products, demos and hoped-for revolutions. Third, they review serious recent trends backed up by serious research, including decreasing labor fraction of income, and increasing wage variance. They vaguely suggest that these trends are caused by computers, but offer relatively little evidence in support of this claim. Finally, they offer a bunch of standard policy recommendations that they would have made anyway, even if all these trends had been the opposite.
While reviewing trends, the book points to this graph (taken from this paper):
It compares recent US productivity growth to growth during the era of electrification, 1890-1940, and suggests that growth might increase soon, if it follows the same pattern. But if this is the growth effect size to expect from computers, it is vastly smaller than the industrial revolution, which sustainably increased growth rates by over a factor of fifty (and is not at all well summarized as caused by steam engines). Of course these book authors are careful not to make strong explicit claims – they are content to vaguely suggest.
So how is one supposed to evaluate a book like this, without original contributions, strong claims, or explicit central arguments to evaluate? The standard intended seems to just be popularity: it is a success if people buy it and mention it lots as they anxiously discuss how computers might change society. And then push for the same policies they would have pushed for anyway, regardless. And by that standard, this book will probably be a success.
I’m planning to write a book, a book I want to both be engaging to a wide audience, and to adequately defend some complex non-obvious intellectual claims. It feels quite daunting to write with both of these goals in mind at once. So I’m thinking of achieving these two goals in two steps. First I’d write a pre-book, which states my main claims and arguments directly and clearly, using expert language, for an expert audience. I’d then circulate that pre-book privately among experts and useful thinkers of various sorts, seeking criticism of my arguments. Then using their feedback, I’d revise my claims and arguments, and write an engaging accessible book that can be circulated widely.
While this strategy seems to make sense, I rarely hear of anyone doing it. Why? Some possible explanations:
The media holds medicine to a lower standard than it holds alternative medicine, such as say crystal healing. No way would an article in a major paper complain that we aren’t subsidizing crystals enough for poor folks, based on the observation that rich folks buy more crystals and rich folks are healthier. But for medicine, that sort of correlation is enough.
“Nearly five black women die needlessly per day from breast cancer” because they don’t have information about the importance of breast screening and they don’t have access to high quality care.
But in fact, the study shows only that across 25 US cities, the ratio of the black vs. white breast cancer death rates correlates (barely significantly) with median city income and a measure of city racial segregation. It is a huge leap to conclude from these correlations that black women don’t have enough info or care!
The very robust health-status correlation predicts more health for higher status folks, and thus more race-health disparity when there is a higher race-status disparity. It seem quite plausible that the race-status disparity is higher in cities where races are segregated and incomes are low.
More from the Post:
It would be nearly nine months before she told herself it was time to act. By then, the lump was the size of a small egg. … Doctors and advocates say the fear that kept her from acting quickly is all too common among black women. It is among the factors that contribute to a disturbing trend: Although they are less likely than white women to get breast cancer, black women are more likely to die from it. … Poverty and racial inequities are the primary factors driving the disparity, according to a study. … The study, which compared mortality rates between black and white women in the nation’s 25 largest cities, states that “nearly five black women die needlessly per day from breast cancer” because they don’t have information about the importance of breast screening and they don’t have access to high quality care. The authors … said genetics play only a small role in the disparity.
More from the study:
[In] the 25 largest cities in the US, … non-Hispanic Black : non-Hispanic White [breast cancer death] rate ratios (RRs) were calculated … Almost all the NHB rates were greater than almost all the NHW rates. … From among the 7 potential correlates, only median household income (r = 0.43, p = 0.037) and a measure of segregation (r = 0.42, p = 0.039) were signiﬁcantly related to the RR.
Note that white women may seem to “get” more breast cancer because they are tested more often for it.
Imagine that a month ago you inherited or won a million dollars. You haven’t spent much, but you did tell people you know and you’ve been thinking about how you will spend it. (Probably including quitting your job.) Today you learn that your favorite pet will die unless you spend a million dollars on medical treatment. Ask yourself: would you spend it? What would most people you know do? In this situation, I’m pretty sure most folks wouldn’t spend a million to save their pet.
Now consider a new Vanity Fair survey:
Questions: Would most people you know kill their favorite pet for $1 million? What about you?
Answers: Most people: Yes (23%) No (72%); Yourself: Yes (11%) No (83%).
I don’t believe it for a minute. Saying you wouldn’t kill your favorite pet for $1 million is cheap talk. Actually declining an offer of $1 million in exchange for the life of your pet, by contrast, costs $1 million. How many people would really turn that offer down in these cash-strapped times?
Actually, my guess is that if no one you knew had ever taken such an offer, and if you took it you’d be in the news so that most folks you know would hear of it, most of you wouldn’t take the offer. But once a few associates had taken the offer, and such offers weren’t newsworthy anymore, most folks would take such offers.
This just shows how much we hate seeming weird. Accepting a million to kill your pet is weird, but then so is paying a million for your pet’s medical treatment. In each case most will do the non-weird thing.
(I posted in July on how you’d take a million to give up the internet.)
In this half-hour episode, Jim Glassman interviews Martin Ford and I on Will Robots Take Our Jobs?. (I posted on Tyler and Ford here.) You might think we’d go into more argument detail in a half hour show, but alas we seem to just repeat the same top level points. This was in part due to an interview, rather than a debate, format. Glassman also seemed more interested in getting Ford to make dramatic claims than in hearing rebuttals – Ford got to say 40% more words than I.
Ideas just aren’t what they used to be. Once upon a time, they could ignite fires of debate, stimulate other thoughts, incite revolutions and fundamentally change the ways we look at and think about the world.
They could penetrate the general culture and make celebrities out of thinkers — notably Albert Einstein, but also Reinhold Niebuhr, Daniel Bell, Betty Friedan, Carl Sagan and Stephen Jay Gould, to name a few. The ideas themselves could even be made famous: for instance, for “the end of ideology,” “the medium is the message,” “the feminine mystique,” “the Big Bang theory,” “the end of history.” A big idea could capture the cover of Time — “Is God Dead?” — and intellectuals like Norman Mailer, William F. Buckley Jr. and Gore Vidal would even occasionally be invited to the couches of late-night talk shows. How long ago that was. If our ideas seem smaller nowadays, it’s not because we are dumber than our forebears but because we just don’t care as much about ideas as they did. …
The real cause may be information itself. … In the past, we collected information … to convert it into … ideas that made sense of the information. … But if information was once grist for ideas, over the last decade it has become competition for them. We are like the farmer who has too much wheat to make flour. We are inundated with so much information that we wouldn’t have time to process it even if we wanted to, and most of us don’t want to. … If a Marx or a Nietzsche were suddenly to appear, blasting his ideas, no one would pay the slightest attention. (more)
While this article adds little to the basic concept, it is a basic concept worth pondering: are big ideas actually less popular today, and if so why? This claim fits with my perception of idea fashion today vs. my memory of thirty years ago, but I have personally changed so much that I don’t trust such memory comparisons.
If this trend is real, I don’t find the “more information” explanation compelling. The amount of available information has been increasing relatively steady for centuries, yet this trend, if real, has only been going for a half century or less. I expect this is more just a long term cycle in intellectual fashion. Once the old established elites get really good at theory, new “young turks” can better make their mark via switching to a fashion where details matter most, and then once those folks are old established elites, there’s a new opening for a fashion favoring theory. Alas for me that, being more a theorist, I happen to reach my peak when theory is most out of fashion.