Friendliness Factors

Imagine several firms competing to make the next generation of some product, like a lawn mower or cell phone.  What factors influence variance in their product quality (relative to cost)?  That is, how much better will the best firm be relative to the average, second best, or worst?   Larger variance factors should make competitors worry more that this round of competition will be their last.  Here are a few factors:

  1. Resource Variance – the more competitors vary in resources, the more performance varies.
  2. Cumulative Advantage – the more prior wins help one win again, the more resources vary.
  3. Grab It First – If the cost to grab and defend a resource is much less than its value, the first to grab can gain a further advantage.
  4. Competitor Count – with more competitors, the best exceeds the second best less, but exceeds the average more. 
  5. Competitor Effort – the longer competitors work before their performance is scored, or the more resources they spend, the more scores vary.
  6. Lumpy Design – the more quality depends on a few crucial choices, relative to many small choices, the more quality varies.
  7. Interdependence – When firms need inputs from each other, winner gains are also supplier gains, reducing variance.   
  8. Info Leaks – the more info competitors can gain about others’ efforts, the more the best will be copied, reducing variance.
  9. Shared Standards – competitors sharing more standards and design features, in info, process, or product, can better understand and use info leaks. 
  10. Legal Barriers – may prevent competitors from sharing standards, info, inputs.
  11. Anti-Trust –  Social coordination may prevent too much winning by a few.
  12. Sharing Deals – If firms own big shares in each other, or form a coop, or just share values, may mind less if others win.  Lets tolerate more variance, but also share more info.
  13. Niche Density – When each competitor can adapt to a different niche, they may all survive.
  14. Quality Sensitivity – demand/success may be very sensitive, or not very sensitive, to quality.
  15. Network Effects – Users may prefer to use the same product regardless of its quality.
  16. [What factors am I missing?  Tell me and I’ll extend the list.]

Some key innovations in history were associated with very high variance in competitor success.  For example, our form of life seems to have eliminated all trace of any other forms on Earth.  On the other hand, farming and industry innovations were associated with much less variance.  I attribute this mainly to info becoming much leakier, in part due to more shared standards, which seems to bode well for our future. 

If you worry that one competitor will severely dominate all others in the next really big innovation, forcing you to worry about its "friendliness," you should want to promote factors that reduce success variance.  (Though if you cared mainly about the winning performance level, you’d want more variance.)

GD Star Rating
Tagged as:
Trackback URL:
  • billswift

    Prices. The more important low costs are to purchasers, the less variance there will be since it will be less profitable to improve the product.

  • If you worry that the next really big innovation will be “unfriendly” in the sense of letting one competitor severely dominate all others

    This simply isn’t the way I use the word “unFriendly”. I use it to refer to terminal values and to final behaviors. A single mind that is more powerful than any other on the playing field, but doesn’t run around killing people or telling them what to do, can be quite Friendly in both the intuitive sense and the benevolent-terminal-values sense.

    Calling this post “Friendliness factors” rather than “Local Vs. Global Takeoff” is needlessly confusing. And I have to seriously wonder – is this the way you had thought I defined “Friendly AI”? If so, this would seem to indicate very little familiarity with my positions, at all.

    Or are you assuming that a superior tactical position automatically equates to “dominant” behavior in the unpleasant sense, hence “unFriendly” in the intuitive sense? This will be true for many possible goal systems, but not ones that have terminal values that assign low utilities to making people unhappy.

  • Eliezer, yes, sorry – I’ve just reworded that sentence.

  • Although perhaps not applicable to many other “races”, when thinking specifically of firms competing to design a new generation of products, an important factor should be the mobility of key personnel among them: this is not just because of (7)-ish “info leaks” effects, since skill and experience also play important parts.

  • What about rent-seeking? I suppose that getting a patent on your drug so that no one else can make it is sort of the opposite of #8, shared standards. On the surface, rent seeking would seem to increase performance variance, more by firms pushing competitors down than by boosting their own product up.

  • Okay, with that rewording – i.e., “These are factors that help determine why, how much, what kind of, and how soon you need to worry about Friendliness” – I agree with all factors you have listed. I would add the following:

    Structure Variance – the more differently designed competitors are, the more they will vary. Behaves much the same way as Resource Variance and may mitigate against Shared Standards.

    Recursivity – the speed at which the “output” of a competitor, in some sense, becomes a resource input or a variant structure.

    These factors and the curve of self-optimization implied in Cumulative Advantage are where I put most of my own attention, and it’s what I think accounts for human brains taking over but Doug Engelbart failing to do so.

    Another factor:

    Shared Values / Smooth Payoffs – the more that “competitors” (which are, in this discussion, being described more like runners in a race than business competitors) share each others’ values, and the more they are thinking in terms of relatively smooth quantitative payouts and less in terms of being the first to reach the Holy Grail, the more likely they are to share info.

    (I.e. this is why Doug Engelbart was more likely to share the mouse with fellow scientists, than AI projects with different values are to cooperate.)

    Others who think about these topics often put their focus on:

    Trustbusting – competitors in aggregate, or a social force outside the set of competitors, try to impose upper limits on power, marketshare, outlaw certain structures, etc. Has subfactors like Monitoring effectiveness, Enforcement effectiveness and speed, etc.

    Ambition – competitors that somehow manage not to want superior positions will probably not achieve them.

    Compacts – competitors that can create and keep binding agreements to share the proceeds of risky endeavors, will be less unequal afterward.

    Reproduction – if successful competitors divide and differentiate they are more likely to create a clade.

    Probably not exhaustive, but that’s what’s coming to mind at the moment.

  • bill shoe

    I think a large influence on variance is whether or not firms want variance. The automotive industry has an active philosophy of avoiding variance. New products are introduced regularly, but they are benchmarked and designed to be as similar to compeititor products as possible. This includes aspects of 6, 7, and 8, but I think it’s stronger than those.

  • Rivalness / Exclusivity – a good design can in principle be used by more than one actor, unless patents prevent it. Vs., one AI that takes over all the poorly defended computing power on the Internet may then defend it against other AIs.

  • Alex, I think of sharing people as a way of sharing info.

    Eliezer, I edited the list to include many of your suggestions. Not sure I understand “recursivity.” I don’t see that AIs have more cumulative advantage than human tool teams, and I suspect this CA concept is better broken into components.

  • frelkins


    Do you count domain knowledge and a flexible culture of innovation as “resources?” If not, I would suggest these as factors.

  • Robin Hanson:

    “If you worry that one competitor will severely dominate all others in the next really big innovation, forcing you to worry about its “friendliness,” you should want to promote factors that reduce success variance.”

    Worrying about Friendliness is orthogonal to whether dominating dynamic is a “single” winner, or a whole economy of competing players. It’s the risk of dominating dynamic doing (irreversible) damage that is the problem. In low-tech environment it’s reduced by having little power, whatever you do with it; in mid-tech environment like ours by human control at all stages, barely sufficient to prevent serious catastrophes; in higher-power environment you’d need a Friendly AI to correctly guide the tech, using carefully calculated planning rather than now-fatal adversarial dynamic. So, reducing the progress towards extremely powerful technology is the only way to postpone the problem of Friendliness (not that I believe at this point it could seriously help).

    Reduced success variance contributes minimally to this effect, and I’m not even sure it contributes in the right direction.

  • Tim Tyler

    I attribute this mainly to info becoming much leakier

    Uh huh. Call us when Google release their page rank algorithm – or when you find out the details of how James Harris Simons handles his investments. That is the type of “leaklessness” we should be concerned with. It seems pretty leakless to me – and I see no sign it is getting “leakier” as time passes.

  • I’m surprised that no one has mentioned network effects, since these have often (recently) led to a single dominant player in some markets (e.g. Microsoft) and also explain the dominance of some standards, practices (e.g. English) and maybe even the genetic code.

    Arguably network effects could be subsumed under (2) but the issue isn’t resources, as (2) states, but enhanced value for customers based on use by other customers.

  • Also I think (7) and (8) (information transfer) are extremely important, as is somewhat suggested by the number of comments related to them already.

    I’d add another related point, modularity, which is essential to allowing offerings to evolve incrementally. This can cut both ways depending on the amount of information transfer.

    There’s a somewhat counter-intuitive pattern here. If a single design becomes dominant, usually but not always through network effects, that greatly enhances information transfer (as implied by (8)). So often a single “firm” becoming totally dominant can create a playing field in which success variance is low due to information transfer.

  • Last comment for now:

    The particular distinctions between “firms”, “products”, “standards” and “innovations” implicit in this post are an artifact of our current social arrangements. In particular the ability of “firms” to internalize “resources” proportionate to their “wins” is very specific to institutions, and varies greatly even within our current economy. Internalization of value created is very low in the internet space, for example.

    Also the post implicitly assumes that the various firms are trying to “win” by competing. Typically firms will try to differentiate, so that they “win” by moving away from their competitors in niche space.

    The result is that very often innovation occurs through finding and dominating new niches rather than through making an existing product better or cheaper to produce. Improving existing products or production methods tends to be due to learning curve effects in which there are many very small innovations, which are typically disseminated throughout the industry fairly quickly.

  • Hyphen

    Tim Tyler: Google’s page rank algorithm *is* released. You can read the reasonably-descriptive patent for it at

    There are also many more, clearer descriptions available if you google “page rank algorithm”

    Of course there are many details that have not been released but the meaty part is public knowledge.

  • Pingback: Overcoming Bias : Emulations Go Foom()

  • Pingback: AI-Foom Debate: Post 1 – 6 | wallowinmaya()