34 Comments

I think it might be more fruitful to better study, in great detail, how distress and pleasures are implemented in the human brain and then create a formalism to look for analogues in other systems.That seems reasonable. At least it gets away from the problemwhere every negative feedback system looks like it has animplicit "goal", and a thermostat with a low setting in a hot roomgets classified as being frustrated...

Expand full comment

What does it mean to talk about the quality of lifeof our visual cortex?I think it's very colorful. ;)

What sized chunks would one even look within for analogs of pleasures and distress?I'm not sure to what degree it is about size, I think it might be more fruitful to better study, in great detail, how distress and pleasures are implemented in the human brain and then create a formalism to look for analogues in other systems.

Expand full comment

Unfortunately, there aren't really existing instances of structuresthat we can point to with human-sized chunks of computationbut much larger communications bandwidths to use as examplesto set our intuition. The closest example are probably pieces ofour brains: What does it mean to talk about the quality of lifeof our visual cortex?

By comparison, an em/upload is an as-close-to-isomorphic mapof a person as technology will allow (if they ever get constructed -but that is a whole separate discussion). I have no problem callingan unmodified em/upload a descendant - but as Hanson himselfhas said, if ems/uploads are set up so that they can modify,replicate, and compete, and therefore evolve, they will rapidlymove away from the current human norm. At that point, even justpicking out what _scale_ of chunk of the computational ecologyto identify with as "a" descendant becomes very arbitrary...What sized chunks would one even look within for analogs ofpleasures and distress?

Expand full comment

Maybe an entity doesn't need to be a person in order to have a "quality of life". If it has even very rough equivalents of human pleasures and distress, that can count at least for utilitarian reasoning.

As for whether or not such beings would be our "decendants", I personally feel inclined to call them that metaphorically, if they are products of what human civilization later becomes or creates. If I'm not mistaken, Robin Hanson would probably consider ems to be our decentants. If it gets very alien and Borg-like, the psychological identification factors obviously decrease, but I personally don't identify with most of humanity terribly much anyway.

Expand full comment

This is kind-of a reply to your Nov 22 post, but I'm putting it hereto distinguish it from the discussion of something more human.

You may be right, in the sense that the dominant (in some sense,the bulk of the mass? energy? computation?) phenotype of along-future civilization is unlikely to be biological homo sapiens.Hanson's original post talked of "descendants", and it is not atall clear that such structures are descendants in any obvious way.

I'm not saying this out of a purely speciesist bias againstuploads/ems. Consider even a very broad view of human thattakes any chunk of computation which is approximately human inscale and approximately human in autonomy to be human.Even _that_ might not exist in an equilibrium evolutionary mix(with no global coordination). Consider our existing communicationstechnology: Optical fibers have been pushed to terabaud rates.A chunk of computing hardware with the equivalent of thestorage and computation parameters of a human brain is likelyto be packaged together with enough communications bandwidthto act more like a lobe of something larger than as an autonomousupload/em.

If that happens, the question of whether "people" arebeing used as cannon fodder seems unanswerable. Nothingleft in the mix looks (computationally) enough like a person forany of our intuitions about quality of life to be meaningful.

Expand full comment

Jeffrey Soreff and gwern, aside from the general relevance of your discussion, the OP was focused on a time when trillions go to war, which will of course be a time that sees AI war drones and rapid artificial reproduction of immediately functional intelligent beings with memories and experiences fully intact. If you talk about a time when both sides in conflict have supersoldier factories, your discussion is not applicable to the original point.

Expand full comment

Touche' Gwern. I am quite aware of the highly fertile subgroups.And yes, I am aware of the general point about evolution makingthe fertile subgroups dominate. Nonetheless:

a) Humans aren't completely at the mercy of differential reproductionrates. Coordination may be hard, but it isn't _impossible_. We canbuild institutions to detect and react to rapidly growing groups.The examples of Japan, South Korea, and Canada in the papersuggest that there is some choice of institutions that continuesto avoid exponential growth, even in the high HDI regime wherethe authors see evidence of fertility increases.

b) The same argument about rapidly growing groups could havebeen made in Malthus's time - and some of the groups havegrowth rates a factor of two per generation above the generalpopulation. There have been enough generations since then forthis to have had a major impact if it were _just_ a matter ofdifferential growth rates. Nonetheless, the population as a whole,including highly fertile subgroups, _hasn't_ bred itself back tothird world subsistence levels. There are pieces missing from thispicture.

c) Yes, I thank you for the link. Note that, except for the singleoutlier at TFI~3 and HDI ~0.91, the data could just as well beinterpreted as saying that the fertility rate goes approximatelyflat at around a TFI of 1.5 above an HDI of ~0.8. Except for theone outlier, the TFI is barely above 2.0 for any of the highdevelopment data points. This isn't an explosion. There isenough time to look for countermeasures before drowning ina tidal wave of human flesh.

Expand full comment

You want data, Soreff, and the general point about evolution is lost on you and you have never heard of highly fertile subgroups like Orthodox Jews or the Amish who have massively increased their numbers even in countries considered to have undergone the demographic transition, and you demand higher level data than that? Well fine, here you go: http://www.nature.com/natur...

Expand full comment

Perhaps you underestimate the possibilities. You blast your enemy's solar system into atoms? But structures can exist in plasma; they might have planned to reconstitute themselves. You plan to chase them down and delete their control systems? But the universe is large; will you pursue them into intergalactic space? I can see conflicts going on for a very long time.

Expand full comment

Pain isn't evolutionarily cheap, even if its implementation is. It has behavioral costs: it prevents otherwise goal-seeking behavior and replaces it with stimulus avoidance. An organism that gives up seeking mates or food because most experiences associated with the search hurts it too much is a dysfunctional phenotype.

Expand full comment

Show me your examples where demographic transitions havebeen reversed. There have been Malthusian arguments forcenturies now, and they've been consistently wrong. Its a badthing to be confident about. The data are against you.

Expand full comment

Everything depends on whether pain is cheap, in evolutionary terms. If by being vulnerable to extreme pain an organism gets a tiny fitness advantage, it will develop that advantage. Evolution doesn't care if you're in pain; it only cares how many copies of you get made. If suffering has little fitness cost or any fitness advantage, then evolution won't economize on pain.

The tendency to think it will is the tendency to imagine that the world is just.

Expand full comment

All the good technology and production comes from concentrations of organisms. When you run away from the population centers you leave the problems AND the benefits. And based on where we see people and other social animals now, it is pretty clear the benefits are much more important than the problems.

Expand full comment

There is, I think, a pretty good chance they can't feel any. The purpose of pain, in evolutionary terms, is to keep an organism alive by concentrating its attention on the pain, the source of the pain, and removing the pain. It seems fairly clear the solution adopted by ants in evolution is "lots of cheap disposable units" rather than "complex, effective, robust, repairable units." To the extent that this analysis is right, there would be no evolutionarly reason for pain mechanisms associated with being dismembered to have been evolved, or to have been maintained in all the evolution that has obviously gone on in the development of the marauder ants.

Expand full comment

We have an extremely hard time predicting what will happen if we put a few thousand off-the-shelf electrical components together in a circuit. What makes you think we have ANY idea what the universe will look like 10^20 years from now? The end of the universe is even wackier to predict than the funner outcomes of global warning. Will it be an infinite expansion to 0 K? Will it be a maximum expansion at which point the universe will fall back in on itself and explode again? Blah blah blah.

At the deepest level, you are putting the model before the reality. There isn't a reason in the world to think that the physics we have developed is any more than rules-of-thumb that appy over the 10 or 20 orders of magnitude we have accessible to us. Thinking to the point of depression that one possible abstraction of some subset of those rules of thumb will apply to our actual universe at some (in my opinion literally) unforseeable future is just a fascinating human bias, that the model is the reality.

Expand full comment

But we don't currently have very effective mind-design tools at our hands. This will almost certainly change in the future; an organizational form that is capable of enabling quadrillions of entities can almost certainly alter motivational structures and perception modules with finesse. All sensory input will be there because it is deliberately chosen to be there.

For instance, entities engaging in physical types of combat will have their motivational structure streamlined to reflect that. They will have no fear of pain or distress, they will not have our reptile brains that override the volitional parts of the frontal lobe because of tissue damage, they won't stop to think how horrible their existence is because the thought will never even occur to them.

Imagine if today, we had the ability to re-design the minds of factory-farmed animals at will. Would we keep them in a state of suffering? No, we would turn them into willing meat drones to be harvested, every single behavoir would be adapted to factory-farming. The suffering stems from a mismatch between modern purpose and legacy mind design.

Expand full comment