Social Innovation Disinterest Puzzle

Back in 1977, I started out college in engineering, then switched to physics, where I got a BS and MS. After that I spent nine years in computer research, at Lockheed and NASA. In physics, engineering, and software I saw that people are quite eager to find better designs, and that the world often pays a lot for them. As a result, it is usually quite hard to find even modesty better designs, at least for devices and mechanisms with modest switching costs.

Over time, I came to notice that many of our most important problems had cores causes in social arrangements. So I started to study economics, and found many simple proposed social innovations that could plausibly lead to large gains. And trying my own hand at looking for innovations, I found more apparently plausible gains. So in 1993 I switched to social science, and started a PhD program at the late age of 34, then having two kids age 0 and 2. (For over a decade after, I didn’t have much free time.)

I naively assumed that the world was just as eager for better social designs. But in fact, the world shows far less interest in better designs for social arrangements. Which, I should have realized, is a better explanation than my unusual genius for why it seemed so easy to find better social designs. But that raises a fundamental puzzle: why does the world seem so much less interested in social innovation, relative to innovation in physical and software devices and systems?

I’ve proposed the thesis of our new book as one explanation. But as many other explanations often come to people’s minds, I thought I might go over why I find them insufficient. Here goes:

Innovation is Hard – Some point to the general fact that innovation is hard. It is hard to know what will work, it is hard to judge effectiveness in a complex world, there are costs to failure, and those who push innovations can seem non-conformist and threatening to status quo powers. But of course these are also obstacles to innovation in physical and software arrangements. We are trying to explain a different attitude toward social innovation.

Social Science Doesn’t Exist – This is basically what I was told by physics profs as an undergrad. Those people in that other building, they just make stuff up, they don’t know anything. And this view is just wrong. We know a whole lot about the social world. We are of course ignorant about a lot of society, but we also retain great ignorance about software and the physical world.

Few Social Gains Possible – Many implicitly assume that social innovation might have been a good idea centuries ago, but today we know nearly all the best ways to arrange firms, employee compensation, rental agreements, marriage contracts, voting systems, and so on. They just can’t imagine future changes in such things remotely as big as past changes, and so assume only small gains are possible. But that’s just crazy; the space of possible social arrangements is vast, and we’ve only explored a few small corners of it.

Humans Have Free Will – Some say that while physics and software can be deterministic, free will means that humans can never be fully predicted. Whatever our best theories predict that humans will do, they can choose to do the opposite, and they often will choose this just to show who’s boss. But in fact we can and do design physical and software systems that accept a lot of uncertainty about the behavior of parts. And we see a lot of pretty predictable behavior in existing social institutions.

Can’t Do Social Experiments – Some say that we can’t test interesting social innovations because it is immoral to do experiments on people if some of them might get worse treatment. But of course we do many medical experiments where some get worse treatment. We tolerate the real world continually producing worse outcomes for many. Paying people to participate in experiments seems to me sufficient to compensate subjects for worse treatment. And in fact experiments are done quite often in the social and human sciences.

Trial Costs Are Too High – Some say the cost of social experiment failure is too high, and point to huge changes like the French Revolution. But just as in physical and software innovation, there’s no excuse for not starting with small trials and tests, and only following with larger trials after smaller-scale success. Big revolutions are mainly a kind of war, not a kind of innovation. Yes war can induce innovation, in both physical and social devices, but surely one can favor innovation without favoring war.

Social Change Can’t Be Designed – Some say that while social systems can improve greatly over time, this can’t or shouldn’t result from conscious deign choices. Design attempts only show human arrogance to think that we can understand the vastly complex social world. And they just encourage those damn regulators to think they can useful interfere in cities and nations. But of course a great many useful social changes have resulted from conscious choices. Including laws and regulations.

Society Is More Complex – Some admit that we know enough about the social world to design useful changes, but claim that in practice we have far fewer occasions to do useful social design because social worlds are far more complex, compared to physical and software worlds. But if we focus on proposed social innovation that are quite simple, with solid arguments for their plausibility, we still find that people are much less interested in exploring or adopting those innovations. In fact, most social mechanisms are far simpler than many widely used physical and software devices.

High Social Switching Costs – Some say that disinterest in social innovation results from it being very expensive to make big changes to our fundamental social institutions. And yes, it is expensive to change things like constitutions or fundamental features of contract law. But we have many proposed social innovations that can be explored at far less cost. We can start with simple math models, simulations, and lab experiments, then try modest field experiments, and finally larger scale field trials. Furthermore, we have often explored and adopted physical and software innovations that were quite expensive to test. Comparing innovations that can be tested at a similar cost, we see far less interest in social innovations.

Value Conflicts – Some say that while we usually agree on which are better physical and software innovations, we have strong value conflicts regarding which are better social arrangements. This makes it hard to trust any advisors on changes, as they can have agendas. But most of our physical and software systems are part of our social systems, and as a result we do usually have differing interests in which physical and software systems are adopted. And advisors regarding physical and software systems can also have strong agendas. Yes, engineers can often agree on technical criteria for evaluating such systems, so that if we make the right transfers between people then adopting the most technically efficient system can be better for everyone. But economist also have constructed similar concepts of efficiency for which similar everyone-can-gain claims can be made.

Misleading Motivations – This is my favored explanation. We are less honest about our motives regarding many familiar social arrangements, relative to physical or software arrangements. We usually point to particular motives as the most important, but those are much less important than we admit. We reward our policy advisors for pretending with us to pursue these usual motives. But we know at some level that we are not that interested in them, and thus also in policy reforms designed to better achieve them. Such reforms don’t solve our real problem. We may show more interest in reforms designed to help us with our real motives, at least if they can let us keep pretending to pursue our pretend motives.

Added 15Jan: The same innovations arouse more interest when they are framed as tech than as social innovations. We see this especially sharply in the blockchain stuff now. Also, the same people are more likely to be hired to work on such innovations if they are presented as tech experts than as social experts. Hedge funds prefer physics over finance PhDs, for example.