Imagine several firms competing to make the next generation of some product, like a lawn mower or cell phone. What factors influence variance in their product quality (relative to cost)? That is, how much better will the best firm be relative to the average, second best, or worst? Larger variance factors should make competitors worry more that this round of competition will be their last. Here are a few factors:
The particular distinctions between "firms", "products", "standards" and "innovations" implicit in this post are an artifact of our current social arrangements. In particular the ability of "firms" to internalize "resources" proportionate to their "wins" is very specific to institutions, and varies greatly even within our current economy. Internalization of value created is very low in the internet space, for example.
Also the post implicitly assumes that the various firms are trying to "win" by competing. Typically firms will try to differentiate, so that they "win" by moving away from their competitors in niche space.
The result is that very often innovation occurs through finding and dominating new niches rather than through making an existing product better or cheaper to produce. Improving existing products or production methods tends to be due to learning curve effects in which there are many very small innovations, which are typically disseminated throughout the industry fairly quickly.
Also I think (7) and (8) (information transfer) are extremely important, as is somewhat suggested by the number of comments related to them already.
I'd add another related point, modularity, which is essential to allowing offerings to evolve incrementally. This can cut both ways depending on the amount of information transfer.
There's a somewhat counter-intuitive pattern here. If a single design becomes dominant, usually but not always through network effects, that greatly enhances information transfer (as implied by (8)). So often a single "firm" becoming totally dominant can create a playing field in which success variance is low due to information transfer.
I'm surprised that no one has mentioned network effects, since these have often (recently) led to a single dominant player in some markets (e.g. Microsoft) and also explain the dominance of some standards, practices (e.g. English) and maybe even the genetic code.
Arguably network effects could be subsumed under (2) but the issue isn't resources, as (2) states, but enhanced value for customers based on use by other customers.
I attribute this mainly to info becoming much leakier
Uh huh. Call us when Google release their page rank algorithm - or when you find out the details of how James Harris Simons handles his investments. That is the type of "leaklessness" we should be concerned with. It seems pretty leakless to me - and I see no sign it is getting "leakier" as time passes.
"If you worry that one competitor will severely dominate all others in the next really big innovation, forcing you to worry about its "friendliness," you should want to promote factors that reduce success variance."Worrying about Friendliness is orthogonal to whether dominating dynamic is a "single" winner, or a whole economy of competing players. It's the risk of dominating dynamic doing (irreversible) damage that is the problem. In low-tech environment it's reduced by having little power, whatever you do with it; in mid-tech environment like ours by human control at all stages, barely sufficient to prevent serious catastrophes; in higher-power environment you'd need a Friendly AI to correctly guide the tech, using carefully calculated planning rather than now-fatal adversarial dynamic. So, reducing the progress towards extremely powerful technology is the only way to postpone the problem of Friendliness (not that I believe at this point it could seriously help).
Reduced success variance contributes minimally to this effect, and I'm not even sure it contributes in the right direction.
Alex, I think of sharing people as a way of sharing info.
Eliezer, I edited the list to include many of your suggestions. Not sure I understand "recursivity." I don't see that AIs have more cumulative advantage than human tool teams, and I suspect this CA concept is better broken into components.
Rivalness / Exclusivity - a good design can in principle be used by more than one actor, unless patents prevent it. Vs., one AI that takes over all the poorly defended computing power on the Internet may then defend it against other AIs.
I think a large influence on variance is whether or not firms want variance. The automotive industry has an active philosophy of avoiding variance. New products are introduced regularly, but they are benchmarked and designed to be as similar to compeititor products as possible. This includes aspects of 6, 7, and 8, but I think it's stronger than those.
Okay, with that rewording - i.e., "These are factors that help determine why, how much, what kind of, and how soon you need to worry about Friendliness" - I agree with all factors you have listed. I would add the following:
Structure Variance - the more differently designed competitors are, the more they will vary. Behaves much the same way as Resource Variance and may mitigate against Shared Standards.
Recursivity - the speed at which the "output" of a competitor, in some sense, becomes a resource input or a variant structure.
These factors and the curve of self-optimization implied in Cumulative Advantage are where I put most of my own attention, and it's what I think accounts for human brains taking over but Doug Engelbart failing to do so.
Another factor:
Shared Values / Smooth Payoffs - the more that "competitors" (which are, in this discussion, being described more like runners in a race than business competitors) share each others' values, and the more they are thinking in terms of relatively smooth quantitative payouts and less in terms of being the first to reach the Holy Grail, the more likely they are to share info.
(I.e. this is why Doug Engelbart was more likely to share the mouse with fellow scientists, than AI projects with different values are to cooperate.)
Others who think about these topics often put their focus on:
Trustbusting - competitors in aggregate, or a social force outside the set of competitors, try to impose upper limits on power, marketshare, outlaw certain structures, etc. Has subfactors like Monitoring effectiveness, Enforcement effectiveness and speed, etc.
Ambition - competitors that somehow manage not to want superior positions will probably not achieve them.
Compacts - competitors that can create and keep binding agreements to share the proceeds of risky endeavors, will be less unequal afterward.
Reproduction - if successful competitors divide and differentiate they are more likely to create a clade.
Probably not exhaustive, but that's what's coming to mind at the moment.
What about rent-seeking? I suppose that getting a patent on your drug so that no one else can make it is sort of the opposite of #8, shared standards. On the surface, rent seeking would seem to increase performance variance, more by firms pushing competitors down than by boosting their own product up.
Although perhaps not applicable to many other "races", when thinking specifically of firms competing to design a new generation of products, an important factor should be the mobility of key personnel among them: this is not just because of (7)-ish "info leaks" effects, since skill and experience also play important parts.
If you worry that the next really big innovation will be "unfriendly" in the sense of letting one competitor severely dominate all others
This simply isn't the way I use the word "unFriendly". I use it to refer to terminal values and to final behaviors. A single mind that is more powerful than any other on the playing field, but doesn't run around killing people or telling them what to do, can be quite Friendly in both the intuitive sense and the benevolent-terminal-values sense.
Calling this post "Friendliness factors" rather than "Local Vs. Global Takeoff" is needlessly confusing. And I have to seriously wonder - is this the way you had thought I defined "Friendly AI"? If so, this would seem to indicate very little familiarity with my positions, at all.
Or are you assuming that a superior tactical position automatically equates to "dominant" behavior in the unpleasant sense, hence "unFriendly" in the intuitive sense? This will be true for many possible goal systems, but not ones that have terminal values that assign low utilities to making people unhappy.
Tim Tyler: Google's page rank algorithm *is* released. You can read the reasonably-descriptive patent for it at http://patft.uspto.gov/neta...
There are also many more, clearer descriptions available if you google "page rank algorithm"
Of course there are many details that have not been released but the meaty part is public knowledge.
Last comment for now:
The particular distinctions between "firms", "products", "standards" and "innovations" implicit in this post are an artifact of our current social arrangements. In particular the ability of "firms" to internalize "resources" proportionate to their "wins" is very specific to institutions, and varies greatly even within our current economy. Internalization of value created is very low in the internet space, for example.
Also the post implicitly assumes that the various firms are trying to "win" by competing. Typically firms will try to differentiate, so that they "win" by moving away from their competitors in niche space.
The result is that very often innovation occurs through finding and dominating new niches rather than through making an existing product better or cheaper to produce. Improving existing products or production methods tends to be due to learning curve effects in which there are many very small innovations, which are typically disseminated throughout the industry fairly quickly.
Also I think (7) and (8) (information transfer) are extremely important, as is somewhat suggested by the number of comments related to them already.
I'd add another related point, modularity, which is essential to allowing offerings to evolve incrementally. This can cut both ways depending on the amount of information transfer.
There's a somewhat counter-intuitive pattern here. If a single design becomes dominant, usually but not always through network effects, that greatly enhances information transfer (as implied by (8)). So often a single "firm" becoming totally dominant can create a playing field in which success variance is low due to information transfer.
I'm surprised that no one has mentioned network effects, since these have often (recently) led to a single dominant player in some markets (e.g. Microsoft) and also explain the dominance of some standards, practices (e.g. English) and maybe even the genetic code.
Arguably network effects could be subsumed under (2) but the issue isn't resources, as (2) states, but enhanced value for customers based on use by other customers.
I attribute this mainly to info becoming much leakier
Uh huh. Call us when Google release their page rank algorithm - or when you find out the details of how James Harris Simons handles his investments. That is the type of "leaklessness" we should be concerned with. It seems pretty leakless to me - and I see no sign it is getting "leakier" as time passes.
Robin Hanson:
"If you worry that one competitor will severely dominate all others in the next really big innovation, forcing you to worry about its "friendliness," you should want to promote factors that reduce success variance."Worrying about Friendliness is orthogonal to whether dominating dynamic is a "single" winner, or a whole economy of competing players. It's the risk of dominating dynamic doing (irreversible) damage that is the problem. In low-tech environment it's reduced by having little power, whatever you do with it; in mid-tech environment like ours by human control at all stages, barely sufficient to prevent serious catastrophes; in higher-power environment you'd need a Friendly AI to correctly guide the tech, using carefully calculated planning rather than now-fatal adversarial dynamic. So, reducing the progress towards extremely powerful technology is the only way to postpone the problem of Friendliness (not that I believe at this point it could seriously help).
Reduced success variance contributes minimally to this effect, and I'm not even sure it contributes in the right direction.
@Robin
Do you count domain knowledge and a flexible culture of innovation as "resources?" If not, I would suggest these as factors.
Alex, I think of sharing people as a way of sharing info.
Eliezer, I edited the list to include many of your suggestions. Not sure I understand "recursivity." I don't see that AIs have more cumulative advantage than human tool teams, and I suspect this CA concept is better broken into components.
Rivalness / Exclusivity - a good design can in principle be used by more than one actor, unless patents prevent it. Vs., one AI that takes over all the poorly defended computing power on the Internet may then defend it against other AIs.
I think a large influence on variance is whether or not firms want variance. The automotive industry has an active philosophy of avoiding variance. New products are introduced regularly, but they are benchmarked and designed to be as similar to compeititor products as possible. This includes aspects of 6, 7, and 8, but I think it's stronger than those.
Okay, with that rewording - i.e., "These are factors that help determine why, how much, what kind of, and how soon you need to worry about Friendliness" - I agree with all factors you have listed. I would add the following:
Structure Variance - the more differently designed competitors are, the more they will vary. Behaves much the same way as Resource Variance and may mitigate against Shared Standards.
Recursivity - the speed at which the "output" of a competitor, in some sense, becomes a resource input or a variant structure.
These factors and the curve of self-optimization implied in Cumulative Advantage are where I put most of my own attention, and it's what I think accounts for human brains taking over but Doug Engelbart failing to do so.
Another factor:
Shared Values / Smooth Payoffs - the more that "competitors" (which are, in this discussion, being described more like runners in a race than business competitors) share each others' values, and the more they are thinking in terms of relatively smooth quantitative payouts and less in terms of being the first to reach the Holy Grail, the more likely they are to share info.
(I.e. this is why Doug Engelbart was more likely to share the mouse with fellow scientists, than AI projects with different values are to cooperate.)
Others who think about these topics often put their focus on:
Trustbusting - competitors in aggregate, or a social force outside the set of competitors, try to impose upper limits on power, marketshare, outlaw certain structures, etc. Has subfactors like Monitoring effectiveness, Enforcement effectiveness and speed, etc.
Ambition - competitors that somehow manage not to want superior positions will probably not achieve them.
Compacts - competitors that can create and keep binding agreements to share the proceeds of risky endeavors, will be less unequal afterward.
Reproduction - if successful competitors divide and differentiate they are more likely to create a clade.
Probably not exhaustive, but that's what's coming to mind at the moment.
What about rent-seeking? I suppose that getting a patent on your drug so that no one else can make it is sort of the opposite of #8, shared standards. On the surface, rent seeking would seem to increase performance variance, more by firms pushing competitors down than by boosting their own product up.
Although perhaps not applicable to many other "races", when thinking specifically of firms competing to design a new generation of products, an important factor should be the mobility of key personnel among them: this is not just because of (7)-ish "info leaks" effects, since skill and experience also play important parts.
Eliezer, yes, sorry - I've just reworded that sentence.
If you worry that the next really big innovation will be "unfriendly" in the sense of letting one competitor severely dominate all others
This simply isn't the way I use the word "unFriendly". I use it to refer to terminal values and to final behaviors. A single mind that is more powerful than any other on the playing field, but doesn't run around killing people or telling them what to do, can be quite Friendly in both the intuitive sense and the benevolent-terminal-values sense.
Calling this post "Friendliness factors" rather than "Local Vs. Global Takeoff" is needlessly confusing. And I have to seriously wonder - is this the way you had thought I defined "Friendly AI"? If so, this would seem to indicate very little familiarity with my positions, at all.
Or are you assuming that a superior tactical position automatically equates to "dominant" behavior in the unpleasant sense, hence "unFriendly" in the intuitive sense? This will be true for many possible goal systems, but not ones that have terminal values that assign low utilities to making people unhappy.
Prices. The more important low costs are to purchasers, the less variance there will be since it will be less profitable to improve the product.