Eliezer yesterday: So really, the whole hard takeoff analysis of “flatline or FOOM” just ends up saying, “the AI will not hit the human timescale keyhole.” From our perspective, an AI will either be so slow as to be bottlenecked, or so fast as to be FOOM. When you look at it that way, it’s not so radical a prediction, is it?
As a consumer and professional, I buy capabilities, not products. I don't buy impressions or features I just might or might not need. I look for the product that most exactly fills all my true needs and nothing more.
I don't care how many funny or awe-inspiring commercials I've seen. If the products do not meet my needs, I won't buy them. If they do, and they're priced reasonably, I will. If a manufacturer makes a product good enough so that it ends up on shopping pages it gets noticed and considered by me. If not, it doesn't exist, and no amount of marketing will change that. It only needs to be available and if it's available I will find it in a meta or site specific search engine, right next to the competition where I can easily choose the one that has the right capabilities.
I admire the sales and advertising industry and derive much pleasure from seeing them perform. I've got a suggestion though: how about just the cold hard facts for a change? Oh, that wouldn't be advertising would it? No salesmanship involved. All your well-honed skills of persuasion would go unused. All your expertise in marketese in vain; you'd have no use for the BS generator module you've got permanently installed in your prefrontal. Needless to say, I find the sales and marketing profession mildly annoying.
If people desire items, such as nuclear bombs, and you've got one, you don't need to sell it, by which I mean spew marketese, place ads and commercials in places where they annoy people. By placing it on the market I mean simply making it available. Just saying, hey, I've got one. The rumor spreads. Hands will be extended with a grabbing motion. This implies there are no competing bomb makers that could grab a market share and leave you with a rotting stockpile. But in case of basic necessities, such as food, regardless of competition there will be demand without you spending a dime on marketing. That's all I'm saying. Just go look at a local (preferably organic) food market. I don't see much advertising for that. I don't care. I need clean food.
Money inflates egos. Science inflates something else. Perhaps even 10x more.
Most of you are missing the point of Paul Graham's essay. It's been a while since I've read it, but if memory serves he was pointing out that determination is needed to get from an idea to a saleable product. That is the hardest part of most creative work - remodeling (which I've done), landscaping design (which I've done even more), writing (which I've worked on), and software (which I've been struggling with). All creative work requires the transition from idea to product. Parenthetically, I've been reading Paul Graham's essays for a couple of years now and I suspect the greatest value Y-Combinator provides is to help keep the startups focused on producing results.
"Our data tells us that technical aptitude - the ability to build a thing - is about 10X less important to the success of a new company than the ability to sell a thing. "
Back in the day, when there were no people able to build things, salesmen thrived?If you have dedicated your life to helping productive people get their products to end users, doesn't mean you're invaluable, or even important, even though you may consider yourself so. You're a replaceable part. A tool. Money is not the problem. Products are the problem. Wikipedia informs there's about 75 trillion of money owned by the bankers. If you're telling me you're doing an invaluable, unique service by placing a wad of money in someone's possession so that they can do what they should be able to anyway, and which you or most of humanity can't, you should look up some history books on what is valuable, to you, to me, to humanity is. Invention, the application of intelligence is. Sales, the application of greed is not. (If you weren't greedy for money but instead for invention, you wouldn't be working in sales and financing.)
If your product fills a natural existing need, there is no need to sell, only to place it on the market. Your only problem is meeting the demand. Humans don't need many products, I for one, need only computing and audio visual equipment (with content) in addition to the basics. Products that need to be sold are products for people who need to fill their empty lives with stuff. If you want to be VERY successful, don't go inventing new needs and finding ways to pushe them on people who don't need them in the first place. Go inventing (better) solutions to fundamental existing problems.
I'm both an investor and an entrepreneur. I've been deeply involved in the start-up space for almost 20 years now. 2 weeks ago we had the annual www.AngelCapitalSummit.org here in Denver, CO USA. This was the largest investor event in the US this year. Note also that it's a *networked* process and event, as it's run on a SaaS platform provided by the www.BusinessCatapult.com. Because of the BC, there were investors and entrepreneurs from around the globe involved w/ the ACS.
So here's what we know after having worked with over a thousand start-ups:
* Nothing moves unless it's sold.* Selling is the hardest thing a new company does.
Our data tells us that technical aptitude - the ability to build a thing - is about 10X less important to the success of a new company than the ability to sell a thing. (Note also that we've yet to find a correlation between success and IP. It appears that much time & $$ is wasted locking up intellectual property that could be better utilized building one's market.) This means that Paul Graham's right. The determination of the founder is key. Entrepreneurs fail until they succeed. The result of this is that successful entrepreneurs are some of the toughest & smartest SOB's in the business ecology.
Want footnotes? Go to the Entrepreneurial Standards Forum (www.ES2F.org) and do the Benchmark Survey. Every entrepreneur that applied to present @ the ACS had to complete this survey before they were considered. From there, ask Jono Shuster, the Exec. Director, if he'd like to talk w/ you about the ESF meta-data.
"Most Internet startups, and the vast, vast majority of small businesses in general, do not require very high intelligence or years of crystallized skill in order to succeed, while AGI requires both. Paul Graham"
Ok... so what? This is one mans opinion and one can argue that there are more then one opinion on the issue. Innovation always requires intelligence that is radical innovation does. I for one have no interest in doing boring uninnovative things and thus I would say determination counts for as much as intelligence.
""I now have enough experience with startups to be able to say what the most important quality is in a startup founder, and it's not what you might think. The most important quality in a startup founder is determination. Not intelligence-- determination." (http://www.paulgraham.com/s...""
This has not been my experience, and to put it in perspective I first tried contracting with government at 19 the idea ultimately went part way but was never adopted that idea was in computers. I am currently starting a new business enterprise which is both pioneering an innovative product as well as dealing with the manufacturing of it and in this current startup I would say that intelligence counts for quite a bit. The math involved is not terribly easy nor are the FEAs required to simulate it.
"What are your achievements in the field? If you want to argue qualifications, you must show at a minimum that 1), qualifications are relevant, and 2), yours are superior to mine."
Well, let's see here AGI is a science, engineering and math problem. You tell me whether qualifications matter? If by qualifications you mean degrees... I can say in my experience degree's in math aren't worth the paper they're written on. If you talk qualifications in terms of intelligence measures like IQ that is also worthless and lest there be any doubt I have an IQ of 180 (only useful if you want into Mensa), so yes I have room to talk. The qualifications that count are having the knowledge and the ability to apply the knowledge to the problem at hand to think outside of the box as it were. For example I never got a bachelors degree I skipped all of that, in fact I taught my self partial differential equations (including all needed prior math), quantum mechanics, theoretical nuclear physics, among other things by the time I was 19. I built an image based motion capture system at 14. I built a hair and cloth simulator for gaming and other real time applications at 13.
As to the second part; what kind of a dumb question is that? All of the things I am working on are proprietary and barring some how getting a signed NDA with you I wouldn't tell you in a million years what I am working on. And if you want to talk further on this issue I suggest that you present your own qualifications for inspection since you were the one making sweeping claims not me.
Robin, what you are calling recursion is what Eliezer referred to as a "cascade".Recursion (what the different definitions have in common) is applying the same formula or method over and over again, usually to its own output.
> "What is wrong with this argument? It is that we have> seen very few million dollar investments ever give> billions in profits."
> Isn't this what the whole venture capital business is> based on? I don't have any statistics for venture> capital investments, but Y Combinator> (http://www.ycombinator.com/) companies routinely sell> for more than a thousand times initial investment.
The statement is technically true... with Y-combinator's typical investments are about $20K and some of their companies do sell for more than 20M.... BUT .......
1. Y Combinator is not a VC firm or even the equivalent of angel investment. Startups typically go to Y-combinator when they are too early stage for either, and with equity pay for both the money they get (small factor) and the "mentorship" (which is the main factor).
2. Selling for 1000 times initial investment means NOTHING. What matters to the investor is how many times initial valuation did it sell? I have heard numbers like $200K for initial valuations with Y-combinator, which as I mentioned earlier is largely since the extra equity pays for "mentoring" that startups receive from Y-combinator.
3. In other words a more realistic valuation for companies that are extremely early stage in angel investments is more like $1M+ for silicon valley startups that manage to get investments. It could be much higher based on the stage of the company and the credentials of the founders.
4. In other words, a necessary condition for a 1000x return on investment is a billion dollar exit (acquisition/merger/IPO), the investors putting in the first round of money when the valuations were low and the founders not having an amazing startup track record before the current startup (since that in turn would inflate the first round valuation, reducing investor margins).
"I take it you have never started a company. Because this set of statements especially "Companies, for one thing, do not require a lot of technical skill to start" is almost completely false in a wide range of circumstances."
Most Internet startups, and the vast, vast majority of small businesses in general, do not require very high intelligence or years of crystallized skill in order to succeed, while AGI requires both. Paul Graham:
"I now have enough experience with startups to be able to say what the most important quality is in a startup founder, and it's not what you might think. The most important quality in a startup founder is determination. Not intelligence-- determination." (http://www.paulgraham.com/s...
"I want to start a startup, but I don't know how to program. How long will it take to learn?
I would guess a smart person can learn to hack sufficiently well in 6 months to a year. The best way to do it would be to find some startup to hire you in an initially menial capacity, and start learning to program on the side." (http://www.paulgraham.com/r...
"This is based on what? Eliezer's impression of the problems difficulty and his lack of a solution?"
The fact that we have, so far, poured thousands of man-years and millions of dollars into the problem with not a single product, or even a working prototype.
"If the second then I would say you haven't done enough in the field to understand what is going on."
What are your achievements in the field? If you want to argue qualifications, you must show at a minimum that 1), qualifications are relevant, and 2), yours are superior to mine.
We can draw useful insights into future innovation from the history of innovation up to this point.
We can also draw useful insights into the modern genetic takeover from the history of previous genetic takeovers. This isn't the first time new heritable media have arisen on the planet.
How much insight these precursors provide can be debated - but IMO, they offer some of the better evidence which we have to work from - ignoring them would make no sense whatsoever.
I understand Eliezer's use of recursion in this context as not just meaning "growth that makes it easier to grow in other ways" but "growth that makes it easier to grow in that very same way." In this sense, none of Robin's comparisons hold up very well. An AGI that can improve its ability to improve its ability is in an extremely tight feedback loop. Things like nuclear fission exhibit this level of tightness, but Robin's many historical examples do not.
It seems to me that the root of the disagreement here is on methodology as much as anything. How do we answer difficult questions of a potentially unprecedented nature?
Robin reasons top-down, from the general to the particular, and he seems to think that the relevant details of this particular question are not unprecendented. He has a strong bias towards reusing specific methodologies (economics, history of science, etc.) that have been used extensively elsewhere. He is intent on finding some way of relating this particular question to previous questions, even if he has to shoehorn it a bit to make it fit. Robin doesn't seem to think that a recursively improving intelligence that has total knowledge of how its mind works is an unprecedented scenario -- at least not so unprecedented that many standard methodologies from elsewhere can't be reused.
Eliezer reasons bottom-up, starting with the phenomenon under question itself, and he believes that the relevant details of this question make it unprecedented. He has a strong bias toward a sort of unstructured Socratic inquiry, or the raw application of intelligence and reason. He starts with the intuition that a recursively improving AGI is sui generis and that the right approach to answering the question involves focusing on the particulars of AGI and abstracting from there, using nothing more than reason to decide what these particulars imply about the answer to the question.
Until they can agree on a methodology to answer the question, I don't think anybody will be swayed. Robin will keep talking about comparable historic events (comparable when seen from an extremely high level and ignoring the particulars) and arguing for the reusing of extant methodologies from other disciplines, and Eliezer will keep focusing on the particulars, insisting that recursively improving AGI is utterly unprecedented, and he'll keep generalizing from particulars of this scenario and showing how no prior recursive event has involved the same particulars.
Rescher is at U Pittsburgh and you can email him; see http://www.pitt.edu/~rescher/. The main Rescher reference would be Scientific Progress: A Philosophical Essay on the Economics of Research in Natural Science. Oxford (Basil Blackwell), 1978. Also see James R. Wible, The Economics of Science (London & New York: Routledge, 1998), which you can preview in Google books and Roland Wagner-Döbler, “Rescher’s Principle of Diminishing Marginal Returns in Scientific Research,” Scientometrics, vol. 50 (2001), pp. 419-36 (which, Robin, you can get thru GMU's online library).
Wagner-Dobler says that Rescher, like Planck, argued that diminishing returns were due to increasing cost of equipment, and so expected it to occur in physics and biology, but not in math. But this can't be right, since Rescher found the same effect in studies of papers on symbolic logic.
A weak point in Rescher's work is his use of the category of "first-rate work". If the quality of work has a distribution, then you can explain some of his results just by saying that the volume of "first-rate work" produced each year is limited by human capacities, and so remains constant. I don't think this is a fatal flaw, because I see confirming evidence other than Rescher's work.
From Wagner-Dobler:
In his book Scientific Progress, Rescher (1978, German ed. 1982, French ed. 1993) developed a principle of decreasing marginal returns of scientific research, which is based, inter-alia, on a law of logarithmic returns and on Lotka's law in a certain interpretation. In the present paper, the historical precursors and the meaning of the principle are sketched out. It is reported on some empirical case studies concerning the principle spread over the literature. New bibliometric data are used about 19th-century mathematics and physics. They confirm Reschers principle apart from the early phases of the disciplines, where a square root law seems to be more applicable. The implication of the principle that the returns of different quality levels grow the slower, the higher the level, is valid. However, the time-derivative ratio between (logarithmized) investment in terms of manpower and returns in terms of first-rate contributors seems not to be linear, but rather to fluctuate vividly, pointing to the cyclical nature of scientific progress. Withregard to Reschers principle, in the light of bibliometric indicators no difference occurs between a natural science like physics and a formal science like mathematics....Not in the same way as the input will expand, however, the amount of first-rate outcomes which grow, according to Rescher, only in a linear manner. Of crucial importance here is the Law of Logarithmic Returns, i.e., the cumulative output of first-rate findings is proportional to the logarithm of cumulative investment. First-rate findings grow in a linear manner as long as investment in science grows exponentially. With the deceleration of investment in science, which began, according to Rescher (and other observers) in the sixties or seventies of our century, a logarithmic retardation of first-rate findings is only a logical consequence. It is the first-rate findings which Rescher identifies with scientific progress, not the sheer mass of inflating standard contributions. And it is not difficult to recognize the implication of Lotka's law that the different levels of importance and quality grow at different rates (see numerical examples at Ref.1, pp. 104-108).
Certain bacteria will grow at an exponential rate as long as there are resources available. The question is not whether minds making minds are capable of rapid expansion. The question should be the nature of the territory into which the mind is expanding. If we're wondering how fast a mind can improve itself, we should ask two types of questions:
1.What is the overall total potential for growth? Is it greatly beyond human cognition, merely more efficient human cognition, or somewhere in between? Is it bounded or infinite?
2.What are the barriers to intellectual development? Does it require vast resource gathering and travel? Are there hurdles and plateaus, or is it continuous?
We all know that exponential growth is possible. In most examples, there turn out to be barriers and caps on how far that growth can go and how fast. This are always properties of the territory into which the growth is expanding.
We need to be asking not whether recursive growth is possible, but how difficult the the territory is for expansion.
As a consumer and professional, I buy capabilities, not products. I don't buy impressions or features I just might or might not need. I look for the product that most exactly fills all my true needs and nothing more.
I don't care how many funny or awe-inspiring commercials I've seen. If the products do not meet my needs, I won't buy them. If they do, and they're priced reasonably, I will. If a manufacturer makes a product good enough so that it ends up on shopping pages it gets noticed and considered by me. If not, it doesn't exist, and no amount of marketing will change that. It only needs to be available and if it's available I will find it in a meta or site specific search engine, right next to the competition where I can easily choose the one that has the right capabilities.
I admire the sales and advertising industry and derive much pleasure from seeing them perform. I've got a suggestion though: how about just the cold hard facts for a change? Oh, that wouldn't be advertising would it? No salesmanship involved. All your well-honed skills of persuasion would go unused. All your expertise in marketese in vain; you'd have no use for the BS generator module you've got permanently installed in your prefrontal. Needless to say, I find the sales and marketing profession mildly annoying.
If people desire items, such as nuclear bombs, and you've got one, you don't need to sell it, by which I mean spew marketese, place ads and commercials in places where they annoy people. By placing it on the market I mean simply making it available. Just saying, hey, I've got one. The rumor spreads. Hands will be extended with a grabbing motion. This implies there are no competing bomb makers that could grab a market share and leave you with a rotting stockpile. But in case of basic necessities, such as food, regardless of competition there will be demand without you spending a dime on marketing. That's all I'm saying. Just go look at a local (preferably organic) food market. I don't see much advertising for that. I don't care. I need clean food.
Money inflates egos. Science inflates something else. Perhaps even 10x more.
Hi Food,
First off, this wasn't my opinion. Though I agree w/ much of what the meta-data from the www.ES2F.org tells us, I'm just reporting it.
Secondarily, re: your statement: "If your product fills a natural existing need, there is no need to sell, only to place it on the market."
Wow...
So where do the markets come from?
Cheers,Kevin
Most of you are missing the point of Paul Graham's essay. It's been a while since I've read it, but if memory serves he was pointing out that determination is needed to get from an idea to a saleable product. That is the hardest part of most creative work - remodeling (which I've done), landscaping design (which I've done even more), writing (which I've worked on), and software (which I've been struggling with). All creative work requires the transition from idea to product. Parenthetically, I've been reading Paul Graham's essays for a couple of years now and I suspect the greatest value Y-Combinator provides is to help keep the startups focused on producing results.
"Our data tells us that technical aptitude - the ability to build a thing - is about 10X less important to the success of a new company than the ability to sell a thing. "
Back in the day, when there were no people able to build things, salesmen thrived?If you have dedicated your life to helping productive people get their products to end users, doesn't mean you're invaluable, or even important, even though you may consider yourself so. You're a replaceable part. A tool. Money is not the problem. Products are the problem. Wikipedia informs there's about 75 trillion of money owned by the bankers. If you're telling me you're doing an invaluable, unique service by placing a wad of money in someone's possession so that they can do what they should be able to anyway, and which you or most of humanity can't, you should look up some history books on what is valuable, to you, to me, to humanity is. Invention, the application of intelligence is. Sales, the application of greed is not. (If you weren't greedy for money but instead for invention, you wouldn't be working in sales and financing.)
If your product fills a natural existing need, there is no need to sell, only to place it on the market. Your only problem is meeting the demand. Humans don't need many products, I for one, need only computing and audio visual equipment (with content) in addition to the basics. Products that need to be sold are products for people who need to fill their empty lives with stuff. If you want to be VERY successful, don't go inventing new needs and finding ways to pushe them on people who don't need them in the first place. Go inventing (better) solutions to fundamental existing problems.
Hi All,
I'm both an investor and an entrepreneur. I've been deeply involved in the start-up space for almost 20 years now. 2 weeks ago we had the annual www.AngelCapitalSummit.org here in Denver, CO USA. This was the largest investor event in the US this year. Note also that it's a *networked* process and event, as it's run on a SaaS platform provided by the www.BusinessCatapult.com. Because of the BC, there were investors and entrepreneurs from around the globe involved w/ the ACS.
So here's what we know after having worked with over a thousand start-ups:
* Nothing moves unless it's sold.* Selling is the hardest thing a new company does.
Our data tells us that technical aptitude - the ability to build a thing - is about 10X less important to the success of a new company than the ability to sell a thing. (Note also that we've yet to find a correlation between success and IP. It appears that much time & $$ is wasted locking up intellectual property that could be better utilized building one's market.) This means that Paul Graham's right. The determination of the founder is key. Entrepreneurs fail until they succeed. The result of this is that successful entrepreneurs are some of the toughest & smartest SOB's in the business ecology.
Want footnotes? Go to the Entrepreneurial Standards Forum (www.ES2F.org) and do the Benchmark Survey. Every entrepreneur that applied to present @ the ACS had to complete this survey before they were considered. From there, ask Jono Shuster, the Exec. Director, if he'd like to talk w/ you about the ESF meta-data.
Cheers,Kevin Johansen, Chairwww.AngelCapitalSummit.org
"Most Internet startups, and the vast, vast majority of small businesses in general, do not require very high intelligence or years of crystallized skill in order to succeed, while AGI requires both. Paul Graham"
Ok... so what? This is one mans opinion and one can argue that there are more then one opinion on the issue. Innovation always requires intelligence that is radical innovation does. I for one have no interest in doing boring uninnovative things and thus I would say determination counts for as much as intelligence.
""I now have enough experience with startups to be able to say what the most important quality is in a startup founder, and it's not what you might think. The most important quality in a startup founder is determination. Not intelligence-- determination." (http://www.paulgraham.com/s...""
This has not been my experience, and to put it in perspective I first tried contracting with government at 19 the idea ultimately went part way but was never adopted that idea was in computers. I am currently starting a new business enterprise which is both pioneering an innovative product as well as dealing with the manufacturing of it and in this current startup I would say that intelligence counts for quite a bit. The math involved is not terribly easy nor are the FEAs required to simulate it.
"What are your achievements in the field? If you want to argue qualifications, you must show at a minimum that 1), qualifications are relevant, and 2), yours are superior to mine."
Well, let's see here AGI is a science, engineering and math problem. You tell me whether qualifications matter? If by qualifications you mean degrees... I can say in my experience degree's in math aren't worth the paper they're written on. If you talk qualifications in terms of intelligence measures like IQ that is also worthless and lest there be any doubt I have an IQ of 180 (only useful if you want into Mensa), so yes I have room to talk. The qualifications that count are having the knowledge and the ability to apply the knowledge to the problem at hand to think outside of the box as it were. For example I never got a bachelors degree I skipped all of that, in fact I taught my self partial differential equations (including all needed prior math), quantum mechanics, theoretical nuclear physics, among other things by the time I was 19. I built an image based motion capture system at 14. I built a hair and cloth simulator for gaming and other real time applications at 13.
As to the second part; what kind of a dumb question is that? All of the things I am working on are proprietary and barring some how getting a signed NDA with you I wouldn't tell you in a million years what I am working on. And if you want to talk further on this issue I suggest that you present your own qualifications for inspection since you were the one making sweeping claims not me.
Robin, what you are calling recursion is what Eliezer referred to as a "cascade".Recursion (what the different definitions have in common) is applying the same formula or method over and over again, usually to its own output.
> "What is wrong with this argument? It is that we have> seen very few million dollar investments ever give> billions in profits."
> Isn't this what the whole venture capital business is> based on? I don't have any statistics for venture> capital investments, but Y Combinator> (http://www.ycombinator.com/) companies routinely sell> for more than a thousand times initial investment.
The statement is technically true... with Y-combinator's typical investments are about $20K and some of their companies do sell for more than 20M.... BUT .......
1. Y Combinator is not a VC firm or even the equivalent of angel investment. Startups typically go to Y-combinator when they are too early stage for either, and with equity pay for both the money they get (small factor) and the "mentorship" (which is the main factor).
2. Selling for 1000 times initial investment means NOTHING. What matters to the investor is how many times initial valuation did it sell? I have heard numbers like $200K for initial valuations with Y-combinator, which as I mentioned earlier is largely since the extra equity pays for "mentoring" that startups receive from Y-combinator.
3. In other words a more realistic valuation for companies that are extremely early stage in angel investments is more like $1M+ for silicon valley startups that manage to get investments. It could be much higher based on the stage of the company and the credentials of the founders.
4. In other words, a necessary condition for a 1000x return on investment is a billion dollar exit (acquisition/merger/IPO), the investors putting in the first round of money when the valuations were low and the founders not having an amazing startup track record before the current startup (since that in turn would inflate the first round valuation, reducing investor margins).
"I take it you have never started a company. Because this set of statements especially "Companies, for one thing, do not require a lot of technical skill to start" is almost completely false in a wide range of circumstances."
Most Internet startups, and the vast, vast majority of small businesses in general, do not require very high intelligence or years of crystallized skill in order to succeed, while AGI requires both. Paul Graham:
"I now have enough experience with startups to be able to say what the most important quality is in a startup founder, and it's not what you might think. The most important quality in a startup founder is determination. Not intelligence-- determination." (http://www.paulgraham.com/s...
"I want to start a startup, but I don't know how to program. How long will it take to learn?
I would guess a smart person can learn to hack sufficiently well in 6 months to a year. The best way to do it would be to find some startup to hire you in an initially menial capacity, and start learning to program on the side." (http://www.paulgraham.com/r...
"This is based on what? Eliezer's impression of the problems difficulty and his lack of a solution?"
The fact that we have, so far, poured thousands of man-years and millions of dollars into the problem with not a single product, or even a working prototype.
"If the second then I would say you haven't done enough in the field to understand what is going on."
What are your achievements in the field? If you want to argue qualifications, you must show at a minimum that 1), qualifications are relevant, and 2), yours are superior to mine.
We can draw useful insights into future innovation from the history of innovation up to this point.
We can also draw useful insights into the modern genetic takeover from the history of previous genetic takeovers. This isn't the first time new heritable media have arisen on the planet.
How much insight these precursors provide can be debated - but IMO, they offer some of the better evidence which we have to work from - ignoring them would make no sense whatsoever.
Peanut, I'll be talking particulars too; I was waiting for that until Eliezer laid out his argument, and then for time to write.
I understand Eliezer's use of recursion in this context as not just meaning "growth that makes it easier to grow in other ways" but "growth that makes it easier to grow in that very same way." In this sense, none of Robin's comparisons hold up very well. An AGI that can improve its ability to improve its ability is in an extremely tight feedback loop. Things like nuclear fission exhibit this level of tightness, but Robin's many historical examples do not.
It seems to me that the root of the disagreement here is on methodology as much as anything. How do we answer difficult questions of a potentially unprecedented nature?
Robin reasons top-down, from the general to the particular, and he seems to think that the relevant details of this particular question are not unprecendented. He has a strong bias towards reusing specific methodologies (economics, history of science, etc.) that have been used extensively elsewhere. He is intent on finding some way of relating this particular question to previous questions, even if he has to shoehorn it a bit to make it fit. Robin doesn't seem to think that a recursively improving intelligence that has total knowledge of how its mind works is an unprecedented scenario -- at least not so unprecedented that many standard methodologies from elsewhere can't be reused.
Eliezer reasons bottom-up, starting with the phenomenon under question itself, and he believes that the relevant details of this question make it unprecedented. He has a strong bias toward a sort of unstructured Socratic inquiry, or the raw application of intelligence and reason. He starts with the intuition that a recursively improving AGI is sui generis and that the right approach to answering the question involves focusing on the particulars of AGI and abstracting from there, using nothing more than reason to decide what these particulars imply about the answer to the question.
Until they can agree on a methodology to answer the question, I don't think anybody will be swayed. Robin will keep talking about comparable historic events (comparable when seen from an extremely high level and ignoring the particulars) and arguing for the reusing of extant methodologies from other disciplines, and Eliezer will keep focusing on the particulars, insisting that recursively improving AGI is utterly unprecedented, and he'll keep generalizing from particulars of this scenario and showing how no prior recursive event has involved the same particulars.
Rescher is at U Pittsburgh and you can email him; see http://www.pitt.edu/~rescher/. The main Rescher reference would be Scientific Progress: A Philosophical Essay on the Economics of Research in Natural Science. Oxford (Basil Blackwell), 1978. Also see James R. Wible, The Economics of Science (London & New York: Routledge, 1998), which you can preview in Google books and Roland Wagner-Döbler, “Rescher’s Principle of Diminishing Marginal Returns in Scientific Research,” Scientometrics, vol. 50 (2001), pp. 419-36 (which, Robin, you can get thru GMU's online library).
Wagner-Dobler says that Rescher, like Planck, argued that diminishing returns were due to increasing cost of equipment, and so expected it to occur in physics and biology, but not in math. But this can't be right, since Rescher found the same effect in studies of papers on symbolic logic.
A weak point in Rescher's work is his use of the category of "first-rate work". If the quality of work has a distribution, then you can explain some of his results just by saying that the volume of "first-rate work" produced each year is limited by human capacities, and so remains constant. I don't think this is a fatal flaw, because I see confirming evidence other than Rescher's work.
From Wagner-Dobler:
In his book Scientific Progress, Rescher (1978, German ed. 1982, French ed. 1993) developed a principle of decreasing marginal returns of scientific research, which is based, inter-alia, on a law of logarithmic returns and on Lotka's law in a certain interpretation. In the present paper, the historical precursors and the meaning of the principle are sketched out. It is reported on some empirical case studies concerning the principle spread over the literature. New bibliometric data are used about 19th-century mathematics and physics. They confirm Reschers principle apart from the early phases of the disciplines, where a square root law seems to be more applicable. The implication of the principle that the returns of different quality levels grow the slower, the higher the level, is valid. However, the time-derivative ratio between (logarithmized) investment in terms of manpower and returns in terms of first-rate contributors seems not to be linear, but rather to fluctuate vividly, pointing to the cyclical nature of scientific progress. Withregard to Reschers principle, in the light of bibliometric indicators no difference occurs between a natural science like physics and a formal science like mathematics....Not in the same way as the input will expand, however, the amount of first-rate outcomes which grow, according to Rescher, only in a linear manner. Of crucial importance here is the Law of Logarithmic Returns, i.e., the cumulative output of first-rate findings is proportional to the logarithm of cumulative investment. First-rate findings grow in a linear manner as long as investment in science grows exponentially. With the deceleration of investment in science, which began, according to Rescher (and other observers) in the sixties or seventies of our century, a logarithmic retardation of first-rate findings is only a logical consequence. It is the first-rate findings which Rescher identifies with scientific progress, not the sheer mass of inflating standard contributions. And it is not difficult to recognize the implication of Lotka's law that the different levels of importance and quality grow at different rates (see numerical examples at Ref.1, pp. 104-108).
Certain bacteria will grow at an exponential rate as long as there are resources available. The question is not whether minds making minds are capable of rapid expansion. The question should be the nature of the territory into which the mind is expanding. If we're wondering how fast a mind can improve itself, we should ask two types of questions:
1.What is the overall total potential for growth? Is it greatly beyond human cognition, merely more efficient human cognition, or somewhere in between? Is it bounded or infinite?
2.What are the barriers to intellectual development? Does it require vast resource gathering and travel? Are there hurdles and plateaus, or is it continuous?
We all know that exponential growth is possible. In most examples, there turn out to be barriers and caps on how far that growth can go and how fast. This are always properties of the territory into which the growth is expanding.
We need to be asking not whether recursive growth is possible, but how difficult the the territory is for expansion.