I guess there must be some sort of register somewhere…

As for the rest of your argument, reasoning is precisely what the normal human does NOT do, and it’s hilarious that you think logical arguments are what keeps most people from theft.

The crux isn’t excluding branches of possible action by an AI, it’s ensuring those avenues aren’t attractive options for any reason.

(agog)

Would you care to explain what the distinction between those two states is?

Yes, it has occurred to ‘them’. I hope you haven’t read http://www.singinst.org/AIRisk.pdf, since if you have, you haven’t grasped the challenge. The crux isn’t excluding branches of possible action by an AI, it’s ensuring those avenues aren’t attractive options for any reason.

This describes monotheism rather well. It has occured to me.

Caledonian – I’d say that one of the key concepts in my current understanding of the Singularity is that it’s the polar opposite of a hard-wired goal. Surely the very idea is that we don’t know what happens inside/beyond a singularity, hence the name?

The whole point of attempting a “Friendly AI” is that its proponents believe that it IS possible to exclude entire branches of possibility from an AI’s courses of action – that the superhuman intelligence can be made safe. Not merely friendly in a human sense, but favorable to human interests, not ‘evil’.

Of course, they cannot provide an objective and rigorous description of what “being in human interests” actually entails, nor can they explain clearly what ‘evil’ is. But they know it when they see it, apparently. And since many of them seem to believe that ‘values’ are arbitrary, they’ve never bothered subjecting what they value to analysis.

Perhaps the possibility that a consequence of an entity being utterly good might be its being utterly unsafe has never occurred to them. And perhaps the possibility that superhuman general intelligence might analyze their values and find them lacking never occurred to them either. That would explain a lot.

That said, I don’t see that a Big Idea About The Future is so different from a Big Idea About The Past in terms of value for humanity. Both can be open or closed, pacifistic or violent, inclusive or exclusive. It’s what you do with it that counts! The question of whether the Singularity as currently defined has positive utility for the human race is not a given, neither will it be unanimous.

I’ve tried and tried, but I can’t think of any other Big Ideas that have stemmed from people looking at where science and technology are going, and extrapolating them to a future point. Perhaps someone who’s less hungover can think of one. Office Christmas do last night, still coming around.

Caledonian – I’d say that one of the key concepts in my current understanding of the Singularity is that it’s the polar opposite of a hard-wired goal. Surely the very idea is that we don’t know what happens inside/beyond a singularity, hence the name?