Overcoming Bias

Share this post

Why Does Software Rot?

www.overcomingbias.com

Discover more from Overcoming Bias

This is a blog on why we believe and do what we do, why we pretend otherwise, how we might do better, and what our descendants might do, if they don't all die.
Over 11,000 subscribers
Continue reading
Sign in

Why Does Software Rot?

Robin Hanson
Jun 23, 2016
Share this post

Why Does Software Rot?

www.overcomingbias.com
32
Share

Almost a year ago computer scientist Daniel Lemire wrote a post critical of a hypothesis I’ve favored, one I’ve used in Age of Em. On the “better late than never” principle, I’ll finally respond now. The hypothesis:

Systems that adapt to contexts tend to get more fragile and harder to readapt to new contexts.

In a 2012 post I said we see this tendency in human brains, in animal brains, in software, in product design, in species, and in individual cells. There is a related academic literature on design feature entrenchment (e.g., here, here, here, here).

Lemire’s 2015 response:

I am arguing back that the open source framework running the Internet, and serving as a foundation for companies like Google and Apple, is a counterexample. Apache, the most important web server software today, is an old piece of technology whose name is a play on words (“a patched server”) indicating that it has been massively patched. The Linux kernel itself runs much of the Internet, and has served as the basis for the Android kernel. It has been heavily updated… Linus Torvalds wrote the original Linux kernel as a tool to run Unix on 386 PCs… Modern-day Linux is thousands of times more flexible.

So we have evolved from writing everything from scratch (in the seventies) to massively reusing and updated pre-existing software. And yet, the software industry is the most flexible, fast-growing industry on the planet. .. If every start-up had to build its own database engine, its own web server… it would still cost millions of dollars to do anything. And that is exactly what would happen if old software grew inflexible: to apply Apache or MySQL to the need of your start-up, you would need to rewrite them first… a costly endeavour. ..

Oracle was not built from the ground up to run on thousands of servers in a cloud environment. So some companies are replacing Oracle with more recent alternatives. But they are not doing so because Oracle has gotten worse, or that Oracle engineers cannot keep up. When I program in Java, I use an API that dates back to 1998 if not earlier. It has been repeatedly updated and it has become more flexible as a result…

Newer programming languages are often interesting, but they are typically less flexible at first than older languages. Everything else being equal, older languages perform better and are faster. They improve over time. .. Just like writers of non-fiction still manage to write large volumes without ending with an incoherent mass, software programmers have learned to cope with very large and very complex endeavours. ..

Programmers, especially young programmers, often prefer to start from scratch. .. In part because it is much more fun to write code than to read code, while both are equally hard. That taste for fresh code is not an indication that starting from scratch is a good habit. Quite the opposite! ..
“Technical debt” .. is a scenario whereas the programmers have quickly adapted to new circumstances, but without solid testing, documentation and design. The software is known to be flawed and difficult, but it is not updated because it “works”. Brains do experience this same effect.

I have long relied on a distinction between architecture and content (see here, here, here, here, here). Content is the part of a system that it is easy to add to or change without changing the rest of the system; architecture is the other part. (Yes, there is a spectrum.) The more content that is fitted to an architecture, and the bigger is that architecture, the harder it becomes to change the architecture.

Lemire’s examples seem to be of systems which grow long and large because they don’t change their core architecture. When an architecture is well enough matched to a stable problem, systems build on it can last long, and grow large, because it is too much trouble to start a competing system from scratch. But when different approaches or environments need different architectures, then after a system grows large enough, one is mostly forced to start over from scratch to use a different enough approach, or to function in a different enough environment.

This is probably why “Some companies are replacing Oracle with more recent alternatives.” Oracle’s architecture isn’t well enough matched. I just can’t buy Lemire’s suggestion that the only reason people ever start new software systems from scratch today is the arrogance and self-indulgence of youth. It happens way far too often to explain that way.

Share this post

Why Does Software Rot?

www.overcomingbias.com
32
Share
32 Comments
Share this discussion

Why Does Software Rot?

www.overcomingbias.com
Stephen Diamond
May 15

Can you think of exceptions?

Exceptions are numerous in philosophy. That's because philosophical differences are the result of conceptual reorganization rather than a gestalt shift in perception. Scientists faced with a new paradigm say things like, 'If that's what physics is going to be, I'm not interested.'

On IQs: top scientists occupy a wide band at the top, say 155 to 190. Declines in fluid intelligence would still put those at the top within the range.

Expand full comment
Reply
Share
gwern
May 15

It would be uncharitable to interpret Planck as giving an exceptionless rule. And I don't know why 'only 5 IQ points' isn't a lot, especially when you pair it with the other age-related declines in WM, energy, short-term memory, sleep quality, and general health; a top scientist only has 10 or 15 points at most over ordinary researchers, so in the QM example, the fiery young turk at age 20 has lost a great deal by the time he's 50 or 60. About the only thing he will have gained is a great deal of experience & knowledge (long-term memory hasn't deteriorated enough to offset the advantage of time), but that's all in the old paradigm, giving him even further incentive to continue working in the old paradigm and ignore or attack any new ones where his cached knowledge is less valuable. Incentives matter a great deal, as does the brain, so this seems adequate to explain Planck's observation without using any metaphors about 'rot'.

Expand full comment
Reply
Share
30 more comments...
Top
New
Community

No posts

Ready for more?

© 2023 Robin Hanson
Privacy ∙ Terms ∙ Collection notice
Start WritingGet the app
Substack is the home for great writing