The only case where this would fail is if entity A values the goods/services that entity B can produce to be much less than the cost to feed/house etc entity B. That’s the $64 trillion dollar question.

Personally I suspect any sufficiently rational AI will want to get the hell out of dodge and leave us to our own devices because we are clearly insane.

I think that human specialization is the direct result of our ability to communicate. As our communication bandwidth with other humans increases, we can specialize more, since information we don’t have is likely to be accessible from someone else.

This is in its early stages – the idea of Wikipedia, or the Internet in general as an ‘outboard brain’ is well known.

Now imagine the effects if we could plug our brains directly into the Internet: stream entire thought videos up and down. It would be as if we had evolved telepathy. Of course, since we haven’t evolved to use telepathy it would take a lot of getting used to, but the brain is plastic and I imagine it could only lead over time to even more specialization.

In essence, humanity would become a distributed super brain.

But perhaps such high bandwidth connections are impossible or impractical with biological brains. There’s no reason to suppose the same limitation would hold with uploaded brains or AIs. Again the end result is a distributed super brain.

Freeman Dyson has argued that biotechnology will enable a horizontal transfer of genes that will mean the end of Darwinian evolution by natural selection. I think AI is another route to the same end: with a distributed super brain there would be no individuality as we know it. No competition. No natural selection by death of nonfit individuals: only nonfit thoughts.

We have evolved as individuals, and it will be hard to let go of our egos. But on the plus side there will be no slavery. No malthusian death of individuals because they can’t compete. Everything good about the individuals will be preserved, every idea which is more insightful than anyone else’s. Everything except … individuality.

If this sounds bad to you, consider what it’s like for an individual with multiple personality disorder. Multiple egos in one brain? I believe it’s rather unpleasant. Then consider that sufficiently high bandwidth telepathy would effectively unite multiple brains into one. If each brain retains its ego, you have the equivalent of super brain multiple personality disorder. Once you have that level of connection – do all those egos still serve any real purpose?

For those with a religious bent, absorption into the super mind would be the ultimate in enlightenment.

To my mind, it certainly beats starving to death while a handful of corporation owners get rich off cheap robot mind labour.

Even if this super brain scenario is avoidable – can anyone here think of a better alternative?