Almost a year ago computer scientist Daniel Lemire wrote a post critical of a hypothesis I’ve favored, one I’ve used in Age of Em. On the “better late than never” principle, I’ll finally respond now. The hypothesis:
Exceptions are numerous in philosophy. That's because philosophical differences are the result of conceptual reorganization rather than a gestalt shift in perception. Scientists faced with a new paradigm say things like, 'If that's what physics is going to be, I'm not interested.'
On IQs: top scientists occupy a wide band at the top, say 155 to 190. Declines in fluid intelligence would still put those at the top within the range.
It would be uncharitable to interpret Planck as giving an exceptionless rule. And I don't know why 'only 5 IQ points' isn't a lot, especially when you pair it with the other age-related declines in WM, energy, short-term memory, sleep quality, and general health; a top scientist only has 10 or 15 points at most over ordinary researchers, so in the QM example, the fiery young turk at age 20 has lost a great deal by the time he's 50 or 60. About the only thing he will have gained is a great deal of experience & knowledge (long-term memory hasn't deteriorated enough to offset the advantage of time), but that's all in the old paradigm, giving him even further incentive to continue working in the old paradigm and ignore or attack any new ones where his cached knowledge is less valuable. Incentives matter a great deal, as does the brain, so this seems adequate to explain Planck's observation without using any metaphors about 'rot'.
I think we're failing to distinguish two phenomena:
1) Architecture compromising its flexibility to support specific content2) Architecture not being flexible enough in the first place
1) happens frequently but not inevitably; it depends on social/political relations between content creators and architecture maintainers. This sort of 'bit rot' happens a lot within companies, but independently-maintained programming languages are much more able to resist it (admittedly, my only citation for this is intuition).
2) eventually happens to any architecture if the world changes enough.
A system fighting 2) can look like one suffering from 1), but the causes are different. Programming generically in C is ugly in a big-ball-of-mud-ish way, but it isn't that way because C designers made ugly hacks to support non-generic programming; it's because it just wasn't designed for generics.
When you say "Oracle’s architecture isn’t well enough matched", that sounds like a case of 2), not of 1).
Given the extreme youth of many of those making QM breakthroughs and the difficulty they had transmitting their insights to others, I am sure that the sheer difficulty of understanding the new paradigm was indeed an issue for many of those whom Planck described in his famous quip (and I've read as much in the memoirs and materials from physicists at the time like Eugene Wigner and other historical accounts). I am not so eager as you to take what was a statistical generalization and applying it to ad hominems against individuals like Einstein. There are always both legitimate and illegitimate grounds for disagreement, and I hope that for Einstein it was more the former than the latter.
Did Einstein reject quantum mechanics because he lacked fluid intelligence? His argument that almost stumped Bohr indicates he retained considerable fluid prowess.
If it were a matter of adequate fluid intelligence, the most eminent scientists would show some proclivity to change paradigms.
There's abundant experimental evidence that mental sets resist change, whereas I don't think it's been clearly established that fluid intelligence facilitates change of opinion. There's even reason to suppose it may make people more stubborn if they're committed to a viewpoint.
Why isn't it easily explained by biological aging causing steep declines in fluid intelligence - typically briefly described as the ability to learn *new* material and paradigms, as opposed to crystallized intelligence - and, of course, ordinary incentives?
Are there any results in geriatrics which indicate that declines in cognitive performance are *not* due to aging? This is a remarkable claim I have never seen before, that cognitive decline with age is not due to the biological underpinnings.
There's the observation that outdated scientific paradigms are replaced only when their users die. This is a decline in flexibility that isn't easily explained by biological aging.
Starting over doesn't mean existing product is displaced though, more that a new one is created. The way to think of it is cockroaches have been around over 100 million years, flexible in some ways, not in others, highly robust at what they do, but not trying to become everything and do everything. Cockroach bit rot is near if not at zero.
Neuronal death with age is enormous. Plaques build up. Bodily integrity and cognitive integrity predict each other. Early childhood intelligence predicts late life intelligence and also longevity, which also are due to genetic correlations. Lots of things go wrong with age. Are there any results in geriatrics which indicate that declines in cognitive performance are *not* due to aging? This is a remarkable claim I have never seen before, that cognitive decline with age is not due to the biological underpinnings. Again, do you think that muscles and liver cells and other cells whose tasks never change decline with aging because of architecture changes...? Where do you get this belief that while all sorts of cells decline in function with age, the brain is a unique special exception to all the usual rules and its simultaneous decline is actually due to an entirely different principle? And I have already explained how nets are very different from human-written software and why they do not seem to rot and why we would expect this from an information-theoretic perspective: tasks share some degree of mutual information and so reuse and adaptation of programs will be more efficient than periodically starting from scratch, and if it isn't, that says more about the badness of the program representations / learning methods than what is possible.
Yes nets can have less architecture, but they still have some. No, it is far from obvious that all human degradation is due to cell aging. Unless you think the systems that nets design are immune from the usual bit rot problems we see in systems humans design, then when nets design their own architectures they should see the same problems. And this predicts mind rot.
I'm not sure what's speculation here. It's a fact that neural nets have relatively minimal architectures compared to hand-coded programs and this is inherent to their power and flexibility, and why they have already surpassed hand-written systems on many problems involving 'complex systems'. Human neural networks are indeed the best learners we know of, but rather than rot constantly, they improve for many decades while being given a wide diversity of task after task, and to the extent human neural networks finally do begin to 'rot', it's directly traceable to the aging process, which makes all biological cells 'rot'. (Do human muscle or liver cells 'rot' through aging because their 'architecture' is pushed too far?) You keep implying that there's some hard and fast distinction between content and architecture and drawing these vast conclusions from this metaphor, but I'm not seeing it. Human written software 'rots' in part because the software is non-adaptive and written in languages in which modified versions are distant and all intermediate versions are broken, which makes it very difficult to evolve or synthesize new programs; deep nets don't rot because the software is highly adaptive and written in a kind of differentiable circuit where intermediate versions still work and the loss surfaces are smooth enough to allow finding new optima, so when the task changes, such as in transfer learning, the new task can be learned with the benefit of all the old knowledge and circuits, and deep net tech advances in part based on making more computations differentiable and allowing switching from jagged surfaces to smoother ones.
PostgreSQL came out at roughly the same time as MySQL. Java was initially released in the mid 90s. So were the languages of choice today like Python and Ruby. The only new language that has managed to make a dent is Go, and there is still no replacement for Java when it comes to scalability.
Humans are the best learners we know of, yet when we try to learn how to build and adapt systems, we run into this problem of increasing fragility as we adapt them. You speculate that your favorite neural networks won't have this problem, but they haven't yet dealt with the kind of complex systems at issue here. I'm very skeptical they will do better.
Can you think of exceptions?
Exceptions are numerous in philosophy. That's because philosophical differences are the result of conceptual reorganization rather than a gestalt shift in perception. Scientists faced with a new paradigm say things like, 'If that's what physics is going to be, I'm not interested.'
On IQs: top scientists occupy a wide band at the top, say 155 to 190. Declines in fluid intelligence would still put those at the top within the range.
It would be uncharitable to interpret Planck as giving an exceptionless rule. And I don't know why 'only 5 IQ points' isn't a lot, especially when you pair it with the other age-related declines in WM, energy, short-term memory, sleep quality, and general health; a top scientist only has 10 or 15 points at most over ordinary researchers, so in the QM example, the fiery young turk at age 20 has lost a great deal by the time he's 50 or 60. About the only thing he will have gained is a great deal of experience & knowledge (long-term memory hasn't deteriorated enough to offset the advantage of time), but that's all in the old paradigm, giving him even further incentive to continue working in the old paradigm and ignore or attack any new ones where his cached knowledge is less valuable. Incentives matter a great deal, as does the brain, so this seems adequate to explain Planck's observation without using any metaphors about 'rot'.
I think we're failing to distinguish two phenomena:
1) Architecture compromising its flexibility to support specific content2) Architecture not being flexible enough in the first place
1) happens frequently but not inevitably; it depends on social/political relations between content creators and architecture maintainers. This sort of 'bit rot' happens a lot within companies, but independently-maintained programming languages are much more able to resist it (admittedly, my only citation for this is intuition).
2) eventually happens to any architecture if the world changes enough.
A system fighting 2) can look like one suffering from 1), but the causes are different. Programming generically in C is ugly in a big-ball-of-mud-ish way, but it isn't that way because C designers made ugly hacks to support non-generic programming; it's because it just wasn't designed for generics.
When you say "Oracle’s architecture isn’t well enough matched", that sounds like a case of 2), not of 1).
It's not presented as a statistical generalization, but as an exceptionless rule.
Anyway, the problem with the fluid-intelligence explanation is that the loss averages only 5 IQ points per decade.
Given the extreme youth of many of those making QM breakthroughs and the difficulty they had transmitting their insights to others, I am sure that the sheer difficulty of understanding the new paradigm was indeed an issue for many of those whom Planck described in his famous quip (and I've read as much in the memoirs and materials from physicists at the time like Eugene Wigner and other historical accounts). I am not so eager as you to take what was a statistical generalization and applying it to ad hominems against individuals like Einstein. There are always both legitimate and illegitimate grounds for disagreement, and I hope that for Einstein it was more the former than the latter.
Did Einstein reject quantum mechanics because he lacked fluid intelligence? His argument that almost stumped Bohr indicates he retained considerable fluid prowess.
If it were a matter of adequate fluid intelligence, the most eminent scientists would show some proclivity to change paradigms.
There's abundant experimental evidence that mental sets resist change, whereas I don't think it's been clearly established that fluid intelligence facilitates change of opinion. There's even reason to suppose it may make people more stubborn if they're committed to a viewpoint.
Why isn't it easily explained by biological aging causing steep declines in fluid intelligence - typically briefly described as the ability to learn *new* material and paradigms, as opposed to crystallized intelligence - and, of course, ordinary incentives?
Are there any results in geriatrics which indicate that declines in cognitive performance are *not* due to aging? This is a remarkable claim I have never seen before, that cognitive decline with age is not due to the biological underpinnings.
There's the observation that outdated scientific paradigms are replaced only when their users die. This is a decline in flexibility that isn't easily explained by biological aging.
Starting over doesn't mean existing product is displaced though, more that a new one is created. The way to think of it is cockroaches have been around over 100 million years, flexible in some ways, not in others, highly robust at what they do, but not trying to become everything and do everything. Cockroach bit rot is near if not at zero.
Neuronal death with age is enormous. Plaques build up. Bodily integrity and cognitive integrity predict each other. Early childhood intelligence predicts late life intelligence and also longevity, which also are due to genetic correlations. Lots of things go wrong with age. Are there any results in geriatrics which indicate that declines in cognitive performance are *not* due to aging? This is a remarkable claim I have never seen before, that cognitive decline with age is not due to the biological underpinnings. Again, do you think that muscles and liver cells and other cells whose tasks never change decline with aging because of architecture changes...? Where do you get this belief that while all sorts of cells decline in function with age, the brain is a unique special exception to all the usual rules and its simultaneous decline is actually due to an entirely different principle? And I have already explained how nets are very different from human-written software and why they do not seem to rot and why we would expect this from an information-theoretic perspective: tasks share some degree of mutual information and so reuse and adaptation of programs will be more efficient than periodically starting from scratch, and if it isn't, that says more about the badness of the program representations / learning methods than what is possible.
Yes nets can have less architecture, but they still have some. No, it is far from obvious that all human degradation is due to cell aging. Unless you think the systems that nets design are immune from the usual bit rot problems we see in systems humans design, then when nets design their own architectures they should see the same problems. And this predicts mind rot.
I'm not sure what's speculation here. It's a fact that neural nets have relatively minimal architectures compared to hand-coded programs and this is inherent to their power and flexibility, and why they have already surpassed hand-written systems on many problems involving 'complex systems'. Human neural networks are indeed the best learners we know of, but rather than rot constantly, they improve for many decades while being given a wide diversity of task after task, and to the extent human neural networks finally do begin to 'rot', it's directly traceable to the aging process, which makes all biological cells 'rot'. (Do human muscle or liver cells 'rot' through aging because their 'architecture' is pushed too far?) You keep implying that there's some hard and fast distinction between content and architecture and drawing these vast conclusions from this metaphor, but I'm not seeing it. Human written software 'rots' in part because the software is non-adaptive and written in languages in which modified versions are distant and all intermediate versions are broken, which makes it very difficult to evolve or synthesize new programs; deep nets don't rot because the software is highly adaptive and written in a kind of differentiable circuit where intermediate versions still work and the loss surfaces are smooth enough to allow finding new optima, so when the task changes, such as in transfer learning, the new task can be learned with the benefit of all the old knowledge and circuits, and deep net tech advances in part based on making more computations differentiable and allowing switching from jagged surfaces to smoother ones.
How one prices, funds and partners a systems is part of its larger social architecture.
PostgreSQL came out at roughly the same time as MySQL. Java was initially released in the mid 90s. So were the languages of choice today like Python and Ruby. The only new language that has managed to make a dent is Go, and there is still no replacement for Java when it comes to scalability.
Humans are the best learners we know of, yet when we try to learn how to build and adapt systems, we run into this problem of increasing fragility as we adapt them. You speculate that your favorite neural networks won't have this problem, but they haven't yet dealt with the kind of complex systems at issue here. I'm very skeptical they will do better.
That IS a good essay.