Someday we may be able to create brain emulations (ems), and someday later we may understand them sufficiently to allow substantial modifications to them. Many have expressed concern that competition for efficient em workers might then turn ems into inhuman creatures of little moral worth. This might happen via reductions of brain systems, features, and activities that are distinctly human but that contribute less to work effectiveness. For example Scott Alexander
> "Upon his death man must leave everything behind ... and depart forever from the world he has known. He must of necessity go to that foul land of death, a fact which makes death the most sorrowful of all events. ... Some foreign doctrines, however, teach that death should not be regarded as profoundly sorrowful. ... These are all gross deceptions contrary to human sentiment and fundamental truths. Not to be happy over happy events, not to be saddened by sorrowful events, not to show surprise at astonishing events, in a word, to consider it proper not to be moved by whatever happens, are all foreign types of deception and falsehood. They are contrary to human nature and extremely repugnant to me."
Motoori Norinaga; or one fun story from _Zen Flesh, Zen Bones_:
> Unsho kept the precepts scrupulously, & Tanzan did as he pleased. One day Unsho visited Tanzan, who was drinking. "Won't you have a drink?" "I never drink!" "One who does not drink is not even human." "Then what am I?" "A Buddha."
These two things could both be true. It could simply be that a little DMN-reduction (a "few moments" once or several times a day, say) is helpful and leaves one's human-ness intact, but as DMN activity approaches zero for a fraction of time approaching 100%, the 'empty shell'-ness effect gets more and more pronounced. Too much of a good thing.
So ems are hypothetical very advanced computer software. Why would anyone want computer software to have moral worth? Isn't it convenient that we can do whatever we want with our software now without worrying that we are killing it or hurting its feelings? It seems like removing the moral worth would be an important step in making ems commercially viable.
One point is that even the most experienced meditator would be nothing compared to an em when it comes to quieting networks that lead to sense of self. Also, quieting the DMN is not uniformly positive.
This leads me to wonder how much hypocrisy there is in people's beliefs about ems' moral value and how much they value tranquility. I value tranquility a lot and expect most em lives to have pressures that would encourage tranquility so their existences seem fine to me, but I wonder how many people hold diverging opinions on these two positions and what their justification for how such divergence is consistent with their views.
Your consideration of the conflict which may be at play is of pertinent interest to me. I've had the unfortunate experience of struggling with bipolar depression recently, and what has been the most tortuous aspect of it has been that I feel trapped, stuck with the DMN. I bounce around my wandering thoughts without surcease. Where you note a normal amount of wandering thoughts may be around 50%, to me it is more like 90%. The inability to actually achieve steady-state focus has been the driver of my negative thoughts.
On this alone (and I acknowledge I present a very biased mindset right now), I wouldn't place too much worry that hyper-focus and clear-mind would lead to loss of our collective self.
Great post, looking forward to the Harris podcast. I guess I'm one of the people who has "expressed concern that competition for efficient em workers might then turn ems into inhuman creatures of little moral worth." However, there are different ways to motivate that worry. Mine is not the worry that we will slice away ems' humanity by some emulated neuro-modification. Actually, I'm rather comforted by that thought. If, like in Peter Watts's Blindsight, the most efficient cognizer completely lacks consciousness, then such ems cannot suffer or dread annihilation.
What scares me is that on some level, those horrible human experiences actually enhance productivity. If this is so, then in the hypercompetitive world, they will not only be retained but amplified. Suppose that at the peak of em capacity to do job x are three almost equally capable person-templates. It might not be the small difference in capability that justifies one of these templates as being the source of most of the running copies. It might just be about which one is willing to work harder and endure more hardship in order to survive. That may very well be the person who most intensely fears non-existence. That fear, if intense enough, might allow this person to forego any human gratification, simply from the recognition that if it slacks off a little bit, it will be replaced by another worker and cease to exist.
I think there is pretty good evidence that the dread of death is common to all higher animals, and that it clearly serves an evolutionary role. Animals more indifferent to the prospect of dying do so at marginally higher rates. Ones who dread it intensely struggle harder to avoid it. Death is omnipresent in the em world in a way it never was in the animal world even in the most Malthusian circumstances: You die simply when another em comes to exist in the world who is better at your job. Dreading this fate is bound to make you up your game a bit. Thus selecting for maximum productivity would ipso facto select for the disposition to fear death to the point of being willing to endure anything to survive. (This mechanic would also select for positive emotions, like loving the work you do, but not exclusively. Basically, since I think that strongly-felt emotions accompany persons of peak productivity, they will not be subtracted in em minds. Also, dread of death is also bound to be computationally simple to emulate; its CPU overhead will certainly be smaller than its power to motivate.)
> "Upon his death man must leave everything behind ... and depart forever from the world he has known. He must of necessity go to that foul land of death, a fact which makes death the most sorrowful of all events. ... Some foreign doctrines, however, teach that death should not be regarded as profoundly sorrowful. ... These are all gross deceptions contrary to human sentiment and fundamental truths. Not to be happy over happy events, not to be saddened by sorrowful events, not to show surprise at astonishing events, in a word, to consider it proper not to be moved by whatever happens, are all foreign types of deception and falsehood. They are contrary to human nature and extremely repugnant to me."
Motoori Norinaga; or one fun story from _Zen Flesh, Zen Bones_:
> Unsho kept the precepts scrupulously, & Tanzan did as he pleased. One day Unsho visited Tanzan, who was drinking. "Won't you have a drink?" "I never drink!" "One who does not drink is not even human." "Then what am I?" "A Buddha."
Ah, I think Sid was making basically the same point.
These two things could both be true. It could simply be that a little DMN-reduction (a "few moments" once or several times a day, say) is helpful and leaves one's human-ness intact, but as DMN activity approaches zero for a fraction of time approaching 100%, the 'empty shell'-ness effect gets more and more pronounced. Too much of a good thing.
So ems are hypothetical very advanced computer software. Why would anyone want computer software to have moral worth? Isn't it convenient that we can do whatever we want with our software now without worrying that we are killing it or hurting its feelings? It seems like removing the moral worth would be an important step in making ems commercially viable.
One point is that even the most experienced meditator would be nothing compared to an em when it comes to quieting networks that lead to sense of self. Also, quieting the DMN is not uniformly positive.
See https://aeon.co/essays/are-... for a careful treatment.
This leads me to wonder how much hypocrisy there is in people's beliefs about ems' moral value and how much they value tranquility. I value tranquility a lot and expect most em lives to have pressures that would encourage tranquility so their existences seem fine to me, but I wonder how many people hold diverging opinions on these two positions and what their justification for how such divergence is consistent with their views.
Your consideration of the conflict which may be at play is of pertinent interest to me. I've had the unfortunate experience of struggling with bipolar depression recently, and what has been the most tortuous aspect of it has been that I feel trapped, stuck with the DMN. I bounce around my wandering thoughts without surcease. Where you note a normal amount of wandering thoughts may be around 50%, to me it is more like 90%. The inability to actually achieve steady-state focus has been the driver of my negative thoughts.
On this alone (and I acknowledge I present a very biased mindset right now), I wouldn't place too much worry that hyper-focus and clear-mind would lead to loss of our collective self.
Great post, looking forward to the Harris podcast. I guess I'm one of the people who has "expressed concern that competition for efficient em workers might then turn ems into inhuman creatures of little moral worth." However, there are different ways to motivate that worry. Mine is not the worry that we will slice away ems' humanity by some emulated neuro-modification. Actually, I'm rather comforted by that thought. If, like in Peter Watts's Blindsight, the most efficient cognizer completely lacks consciousness, then such ems cannot suffer or dread annihilation.
What scares me is that on some level, those horrible human experiences actually enhance productivity. If this is so, then in the hypercompetitive world, they will not only be retained but amplified. Suppose that at the peak of em capacity to do job x are three almost equally capable person-templates. It might not be the small difference in capability that justifies one of these templates as being the source of most of the running copies. It might just be about which one is willing to work harder and endure more hardship in order to survive. That may very well be the person who most intensely fears non-existence. That fear, if intense enough, might allow this person to forego any human gratification, simply from the recognition that if it slacks off a little bit, it will be replaced by another worker and cease to exist.
I think there is pretty good evidence that the dread of death is common to all higher animals, and that it clearly serves an evolutionary role. Animals more indifferent to the prospect of dying do so at marginally higher rates. Ones who dread it intensely struggle harder to avoid it. Death is omnipresent in the em world in a way it never was in the animal world even in the most Malthusian circumstances: You die simply when another em comes to exist in the world who is better at your job. Dreading this fate is bound to make you up your game a bit. Thus selecting for maximum productivity would ipso facto select for the disposition to fear death to the point of being willing to endure anything to survive. (This mechanic would also select for positive emotions, like loving the work you do, but not exclusively. Basically, since I think that strongly-felt emotions accompany persons of peak productivity, they will not be subtracted in em minds. Also, dread of death is also bound to be computationally simple to emulate; its CPU overhead will certainly be smaller than its power to motivate.)