understanding low level brain processes enough to aid em corner-cutting need not help much with understanding high level architecture.

Certainly this could be true given what we know now, but I’m pretty confident that it is unlikely, based on a fairly large number of examples of how people are trying and the tools they need.

I guess to you it seems likely but I don’t know why.

If we want to pursue this probably the only way to pin down where we diverge is to get into the specifics of how we judge where the probability mass is in this domain. I can’t do that right now but I’m willing if you want to later.

When writing my earlier response I was thinking we hadn’t produced anything in the computing and AI domain comparable to the Heisenberg uncertainty principle, Newton’s laws, etc. And perhaps we haven’t. But we have produced some insights that rise well above “just engineering”.

Notably most of these insights are quite directly traceable to engineers working on a large set of related problems for decades, and sometimes beating their heads against a wall that the insight finally made visible. Note that many of the insights are negative.

Here’s a quick sampling because I don’t have time to elaborate. Maybe we can discuss later if people want.

• Information as a measurable quantity.
• Turing’s incomputability results
• Complexity hierarchy, and intractability proofs for various flavors of reasoning and search
• Search and optimization as basic elements of AI systems
• Komolgorov entropy / maximum entropy / minimum description length
• Switch from logic to statistical learning as the conceptual language of AI
• Use of population / evolutionary methods and analysis

So I agree with Eliezer and Don that insight is required. I think if we had tried to just “muddle through” without these insights we’d be progressing very slowly, if at all.

Conversely however I think that insight generally comes from accumulated engineering examples (successful and unsuccessful) that outline the issue to be understood, the way flour in the air of a garage can show what invisible animal is present (if any).

So after reflection if we have any disagreement, it is about how to get to insight.

In short, there are processes which acquire knowledge at vastly different efficiencies and even the most efficient one we know of shows many flaws. So is it really fantasy that it might be possible for something to build something which acquires the information much faster?

None of the above.

A young human is not just a passive recipient of data, but is interacting with it. It’s the interactions that are largely responsible for grow in the human’s intelligence.

Experiments with kittens have demonstrated that interaction is important: http://books.google.com/books?id=etNd-4ppKv8C&pg=PA54&lpg=PA54&dq=kitten+visual+stimulation+experiment&source=web&ots=MiURgJJ4vB&sig=fuUVQewVRdKMScNQRXRISPyM_T8#PPA54,M1

the internet doesn’t really count as content to creatures that don’t know how to parse it and use it in reasoning.

The idea is that organisms can learn. Like babies learn. You need some content to be a baby in the first place – but it doesn’t seem to be an enormous quantity.

Jed, understanding low level brain processes enough to aid em corner-cutting need not help much with understanding high level architecture.

Marcello, yes of course feeding raw data into the right architecture could eventually produce human level intelligence; I meant it is fantasy to think this could take a reasonable time, relative to the option of making use of the content human minds now hold, which is our precious heritage.

Eliezer, yes well-chosen priors are the key “encoded info.” There may be a misunderstanding that when I say “info” people think I mean direct facts like “Paris is capital of France”, while I instead mean any content within your architecture that helps you focus attention well. Clearly human babies do leave out Bayes’ Rule and modus ponens, but yes we should put that in if we can cleanly do so. I’d just claim that doesn’t get you very far; you’ll need to find a way to inherit big chunks of the vast human content heritage.

“But only AIs in the second class can be knowably Friendly, and I suspect that the proportion of worlds that survive the first type of AI development is tiny.”

I guess most of those who disagree with you would welcome some sort of less vague explanation of both premises or perhaps some proofs.

First, as you correctly state, we know that it is possible to build AI without insight. OTOH, there is nothing to support “insight” path.

Second, it looks like humans are built without insight, but still are generally friendly to fellow humans. It looks like the key is that there is many other humans involved in the environment. It also appears that as intelligence grows, we generally tend to be MORE friendly to our fellow humans.

So to sum it up, we know that a lot of minds created by blind process without insight seems to be quite friendly to other minds.

What you seem to propose is that the only possible path leading to friendly AI is SINGLE mind created WITH INSIGHT. That is the EXACT OPPOSITE of what we know to work.

And the only argument to support your thesis is the recursion theory – despite the fact that many of us see human civilization already living in tight recursion environment.

You should not be surprised that some of us consider your theory somewhat ridiculous.

“You need content – but we have a whole internet of content, mostly available for anyone – though of course only Google has access to certain important resources – such as Google Books. More than content, you need actuators that affect a world, and feedback about which actions are effective. For Google the actuators are its search results – and the feedback it gets consists of who clicks on which link. For traders, the actuators are investments, and the stock price provides feedback.”

Correct. Indeed, mining internet content for knowledge is the obvious way how to start.

Anyway, I believe that existing knowldge bases like Cyc or OpenCog can provide a good feedback that your “AI miner” gets the correct knowledge – and much faster than anything that involves human interaction.

If you can develop algorithm that gets the same results just by scanning arbitrary text in internet as those that are hardcoded Cyc database, you are half-way there…