<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
		>
<channel>
	<title>Comments on: Total Tech Wars</title>
	<atom:link href="http://www.overcomingbias.com/2008/11/total-tech-wars.html/feed" rel="self" type="application/rss+xml" />
	<link>http://www.overcomingbias.com/2008/11/total-tech-wars.html</link>
	<description>Overcoming Bias is economist Robin Hanson’s blog, on honesty, signaling, disagreement, forecasting, and the far future.</description>
	<lastBuildDate>Sat, 11 Feb 2012 23:23:58 +0000</lastBuildDate>
	<sy:updatePeriod>hourly</sy:updatePeriod>
	<sy:updateFrequency>1</sy:updateFrequency>
	<generator>http://wordpress.org/?v=3.1.3</generator>
	<item>
		<title>By: Ely Spears</title>
		<link>http://www.overcomingbias.com/2008/11/total-tech-wars.html#comment-478985</link>
		<dc:creator>Ely Spears</dc:creator>
		<pubDate>Mon, 27 Jun 2011 00:16:26 +0000</pubDate>
		<guid isPermaLink="false">http://prod.ob.trike.com.au/2008/11/total-tech-wars.html#comment-478985</guid>
		<description>I think in the discussion above, there is a lot of conflation between causes for what is termed &quot;total tech war.&quot; You can find yourself in a total tech war merely by believing that the other agents see it as such. Or you can independently analyze the situation and determine that the best way to maximize your own payoff is to treat it as a total tech war regardless of what the other agents will think about it. If the space of advantages as upward cliffs as Eliezer suggests, then it is not unreasonable to believe an agent with a time sensitive, but utterly dominating, advantage will rationally decide the most payoff happens by acting in accordance with a total tech war plan of action. This is especially true if part of the cliff-advantage is the ability to analyze a situation more deeply and rapidly than competitors. I don&#039;t see any reason why extra, special arguments are needed to justify this as a realistic scenario within AI FOOM.</description>
		<content:encoded><![CDATA[<p>I think in the discussion above, there is a lot of conflation between causes for what is termed &#8220;total tech war.&#8221; You can find yourself in a total tech war merely by believing that the other agents see it as such. Or you can independently analyze the situation and determine that the best way to maximize your own payoff is to treat it as a total tech war regardless of what the other agents will think about it. If the space of advantages as upward cliffs as Eliezer suggests, then it is not unreasonable to believe an agent with a time sensitive, but utterly dominating, advantage will rationally decide the most payoff happens by acting in accordance with a total tech war plan of action. This is especially true if part of the cliff-advantage is the ability to analyze a situation more deeply and rapidly than competitors. I don&#8217;t see any reason why extra, special arguments are needed to justify this as a realistic scenario within AI FOOM.</p>
]]></content:encoded>
	</item>
	<item>
		<title>By: Tim Tyler</title>
		<link>http://www.overcomingbias.com/2008/11/total-tech-wars.html#comment-391669</link>
		<dc:creator>Tim Tyler</dc:creator>
		<pubDate>Sun, 30 Nov 2008 17:33:39 +0000</pubDate>
		<guid isPermaLink="false">http://prod.ob.trike.com.au/2008/11/total-tech-wars.html#comment-391669</guid>
		<description>&lt;BLOCKQUOTE&gt;&lt;EM&gt;if you want to say that a particular tech is more winner take all than usual, you need an argument based on more than just this effect. And if you want to argument it is far more so than any other tech humans have ever seen, you need a damn good additional argument.&lt;/EM&gt;&lt;/BLOCKQUOTE&gt;

IT is the bleeding edge of technology - and is more effective than most tech at creating inequalities - e.g. look at the list of top billionaires.

Machine intelligence is at the bleeding edge of IT.  It is IT&#039;s &quot;killer application&quot;.  Whether its inventors will exploit its potential to provide wealth will be a matter of historical contingency - but the potential certainly looks as though it will be there. In particular, it looks as though it is likely to be mostly a server-side technology - and those are the easiest for the owners to hang on to - by preventing others from reverse-engineering the technology.

</description>
		<content:encoded><![CDATA[<blockquote><p><em>if you want to say that a particular tech is more winner take all than usual, you need an argument based on more than just this effect. And if you want to argument it is far more so than any other tech humans have ever seen, you need a damn good additional argument.</em></p></blockquote>
<p>IT is the bleeding edge of technology &#8211; and is more effective than most tech at creating inequalities &#8211; e.g. look at the list of top billionaires.</p>
<p>Machine intelligence is at the bleeding edge of IT.  It is IT&#8217;s &#8220;killer application&#8221;.  Whether its inventors will exploit its potential to provide wealth will be a matter of historical contingency &#8211; but the potential certainly looks as though it will be there. In particular, it looks as though it is likely to be mostly a server-side technology &#8211; and those are the easiest for the owners to hang on to &#8211; by preventing others from reverse-engineering the technology.</p>
]]></content:encoded>
	</item>
	<item>
		<title>By: James Andrix</title>
		<link>http://www.overcomingbias.com/2008/11/total-tech-wars.html#comment-391668</link>
		<dc:creator>James Andrix</dc:creator>
		<pubDate>Sun, 30 Nov 2008 16:47:36 +0000</pubDate>
		<guid isPermaLink="false">http://prod.ob.trike.com.au/2008/11/total-tech-wars.html#comment-391668</guid>
		<description>Ian:
What benefits could slaves or free men provide an AI (or a group of EMs) that it could do itself 100 times better with nanobots and new processor cores. Foxes do not enslave rabbits.
Even if it were just deciding to enslave us or let us be &#039;free&#039;, it would know that it could always free us later. (us or future generations)

General Thinker:
A simulated human can be put in a simulated environment that is sped up right along with them. They do a year&#039;s worth of cognitive work, and play, and sleep, in hours. They experience the full year.
</description>
		<content:encoded><![CDATA[<p>Ian:<br />
What benefits could slaves or free men provide an AI (or a group of EMs) that it could do itself 100 times better with nanobots and new processor cores. Foxes do not enslave rabbits.<br />
Even if it were just deciding to enslave us or let us be &#8216;free&#8217;, it would know that it could always free us later. (us or future generations)</p>
<p>General Thinker:<br />
A simulated human can be put in a simulated environment that is sped up right along with them. They do a year&#8217;s worth of cognitive work, and play, and sleep, in hours. They experience the full year.</p>
]]></content:encoded>
	</item>
	<item>
		<title>By: Tyrrell McAllister</title>
		<link>http://www.overcomingbias.com/2008/11/total-tech-wars.html#comment-391667</link>
		<dc:creator>Tyrrell McAllister</dc:creator>
		<pubDate>Sun, 30 Nov 2008 16:04:24 +0000</pubDate>
		<guid isPermaLink="false">http://prod.ob.trike.com.au/2008/11/total-tech-wars.html#comment-391667</guid>
		<description>@GenericThinker
&lt;blockquote&gt;&lt;em&gt;Unless you plan to build an AI and immediately hook it up to all your nuclear weapons or give it free run of any and all factories for making weapons or nanotech (whatever that is nano being applied to all manner of things and now being essentially a buzz word) there is no reason to assume friendly or not an AI would destroy the world. That entire scenario is fictional, one can build an AI in an isolated computer system with little to no danger.
. . .
As long as one keeps a human in the loop with the AI and properly designs the hardware and what the hardware is connected to one can make a safe trip into &quot;mind design space&quot;.
&lt;/em&gt;&lt;/blockquote&gt;

Eliezer has made a &lt;a href=http://yudkowsky.net/singularity/aibox rel=&quot;nofollow&quot;&gt;very persuasive case&lt;/a&gt; that a &quot;human in the loop&quot; would not be adequate to contain a super-intelligent AI.  See also his post &lt;a href=http://www.overcomingbias.com/2008/05/faster-than-ein.html rel=&quot;nofollow&quot;&gt;That Alien Message&lt;/a&gt;.  This post also addresses the point you raise when you write, &quot;A final note there is this continued talk of 100&#039;s or 1000&#039;s of times human speed what does that even mean? What would thinking at a thousand times our current speed mean? How would that even work? Our thoughts and senses etc are tied closely to our sense of time. If you speed that up why would one expect that would be better? Seems to me that this would merely be like listening to music on fast-forward.&quot;

Neither of these addresses the question of whether an artificial super-intelligence is likely to be built in the near future.  But they might make you reconsider your expectation that it could be easily controlled if it was built.
</description>
		<content:encoded><![CDATA[<p>@GenericThinker</p>
<blockquote><p><em>Unless you plan to build an AI and immediately hook it up to all your nuclear weapons or give it free run of any and all factories for making weapons or nanotech (whatever that is nano being applied to all manner of things and now being essentially a buzz word) there is no reason to assume friendly or not an AI would destroy the world. That entire scenario is fictional, one can build an AI in an isolated computer system with little to no danger.<br />
. . .<br />
As long as one keeps a human in the loop with the AI and properly designs the hardware and what the hardware is connected to one can make a safe trip into &#8220;mind design space&#8221;.<br />
</em></p></blockquote>
<p>Eliezer has made a <a href=http://yudkowsky.net/singularity/aibox rel="nofollow">very persuasive case</a> that a &#8220;human in the loop&#8221; would not be adequate to contain a super-intelligent AI.  See also his post <a href=http://www.overcomingbias.com/2008/05/faster-than-ein.html rel="nofollow">That Alien Message</a>.  This post also addresses the point you raise when you write, &#8220;A final note there is this continued talk of 100&#8242;s or 1000&#8242;s of times human speed what does that even mean? What would thinking at a thousand times our current speed mean? How would that even work? Our thoughts and senses etc are tied closely to our sense of time. If you speed that up why would one expect that would be better? Seems to me that this would merely be like listening to music on fast-forward.&#8221;</p>
<p>Neither of these addresses the question of whether an artificial super-intelligence is likely to be built in the near future.  But they might make you reconsider your expectation that it could be easily controlled if it was built.</p>
]]></content:encoded>
	</item>
	<item>
		<title>By: Robin Hanson</title>
		<link>http://www.overcomingbias.com/2008/11/total-tech-wars.html#comment-391666</link>
		<dc:creator>Robin Hanson</dc:creator>
		<pubDate>Sun, 30 Nov 2008 13:27:10 +0000</pubDate>
		<guid isPermaLink="false">http://prod.ob.trike.com.au/2008/11/total-tech-wars.html#comment-391666</guid>
		<description>Eliezer, if everything is at stake then &quot;winner take all&quot; is &quot;total war&quot;; it doesn&#039;t really matter if they shoot your or just starve you to death.  The whole point of this post is to note that anything can be seen as &quot;winner-take-all&quot; just by expecting others to see it that way.  So if you want to say that a particular tech is &lt;i&gt;more&lt;/i&gt; winner take all than usual, you need an argument based on more than just this effect.   And if you want to argument it is &lt;i&gt;far&lt;/i&gt; more so than any other tech humans have ever seen, you need a damn good additional argument.  It is possible that you could make such an argument work based on the &quot;tech landscape&quot; considerations you mention, but I haven&#039;t seen that yet.  So consider this post to be yet another reminder that I await hearing your core argument; until then I set the stage with posts like this.

To answer your direct questions, I am not suggesting forbidding speaking of anything, and if &quot;unfriendly AI&quot; is &lt;i&gt;defined&lt;/i&gt; as an AI who sees itself in a total war, then sure it would take a total war strategy of fighting not trading.  But you haven&#039;t actually defined &quot;unfriendly&quot; yet.

Carl, I replied at your blog post, but will repeat my point here this time.  You say total war doesn&#039;t happen now because leaders are &quot;comfortable&quot; and humans are risk-averse with complex preferences, but there would be a total war over the solar system later because evolved machines preferences would be linear in raw materials. But evolutionary arguments don&#039;t say we evolve to only care about raw materials, and they only suggest risk-neutrality with respect to fluctuations that are largely independent across copies that share &quot;genes.&quot;  With respect to correlated fluctuations evolutionary arguments suggest risk-averse log utility.
</description>
		<content:encoded><![CDATA[<p>Eliezer, if everything is at stake then &#8220;winner take all&#8221; is &#8220;total war&#8221;; it doesn&#8217;t really matter if they shoot your or just starve you to death.  The whole point of this post is to note that anything can be seen as &#8220;winner-take-all&#8221; just by expecting others to see it that way.  So if you want to say that a particular tech is <i>more</i> winner take all than usual, you need an argument based on more than just this effect.   And if you want to argument it is <i>far</i> more so than any other tech humans have ever seen, you need a damn good additional argument.  It is possible that you could make such an argument work based on the &#8220;tech landscape&#8221; considerations you mention, but I haven&#8217;t seen that yet.  So consider this post to be yet another reminder that I await hearing your core argument; until then I set the stage with posts like this.</p>
<p>To answer your direct questions, I am not suggesting forbidding speaking of anything, and if &#8220;unfriendly AI&#8221; is <i>defined</i> as an AI who sees itself in a total war, then sure it would take a total war strategy of fighting not trading.  But you haven&#8217;t actually defined &#8220;unfriendly&#8221; yet.</p>
<p>Carl, I replied at your blog post, but will repeat my point here this time.  You say total war doesn&#8217;t happen now because leaders are &#8220;comfortable&#8221; and humans are risk-averse with complex preferences, but there would be a total war over the solar system later because evolved machines preferences would be linear in raw materials. But evolutionary arguments don&#8217;t say we evolve to only care about raw materials, and they only suggest risk-neutrality with respect to fluctuations that are largely independent across copies that share &#8220;genes.&#8221;  With respect to correlated fluctuations evolutionary arguments suggest risk-averse log utility.</p>
]]></content:encoded>
	</item>
	<item>
		<title>By: Tim Tyler</title>
		<link>http://www.overcomingbias.com/2008/11/total-tech-wars.html#comment-391665</link>
		<dc:creator>Tim Tyler</dc:creator>
		<pubDate>Sun, 30 Nov 2008 10:22:42 +0000</pubDate>
		<guid isPermaLink="false">http://prod.ob.trike.com.au/2008/11/total-tech-wars.html#comment-391665</guid>
		<description>&lt;BLOCKQUOTE&gt;&lt;EM&gt;A winner-take-all scenario doesn&#039;t imply destructive combat or any sort of military conflict.&lt;/EM&gt;&lt;/BLOCKQUOTE&gt;

Just so. For example, in biology, genes can come to dominate completely by differential reproductive success - not just by killing all your competitors.  Warfare is different from winning.
</description>
		<content:encoded><![CDATA[<blockquote><p><em>A winner-take-all scenario doesn&#8217;t imply destructive combat or any sort of military conflict.</em></p></blockquote>
<p>Just so. For example, in biology, genes can come to dominate completely by differential reproductive success &#8211; not just by killing all your competitors.  Warfare is different from winning.</p>
]]></content:encoded>
	</item>
	<item>
		<title>By: Cameron Taylor</title>
		<link>http://www.overcomingbias.com/2008/11/total-tech-wars.html#comment-391664</link>
		<dc:creator>Cameron Taylor</dc:creator>
		<pubDate>Sun, 30 Nov 2008 08:38:48 +0000</pubDate>
		<guid isPermaLink="false">http://prod.ob.trike.com.au/2008/11/total-tech-wars.html#comment-391664</guid>
		<description>Will underestimates the competitive potential of the Trifid.
</description>
		<content:encoded><![CDATA[<p>Will underestimates the competitive potential of the Trifid.</p>
]]></content:encoded>
	</item>
	<item>
		<title>By: Cameron Taylor</title>
		<link>http://www.overcomingbias.com/2008/11/total-tech-wars.html#comment-391663</link>
		<dc:creator>Cameron Taylor</dc:creator>
		<pubDate>Sun, 30 Nov 2008 08:35:34 +0000</pubDate>
		<guid isPermaLink="false">http://prod.ob.trike.com.au/2008/11/total-tech-wars.html#comment-391663</guid>
		<description>There may be, as Robin notes, a bias towards seeing new tech through the lens of total war. There is also a bias for optimism, based off mellenia of shot messengers and a bias to see the future through the lens of the present.

I see in this post a warning against mentioning potentially disasterous consequences to AI. This scares me. When even rationalists ought to feel ashamed of considering such possibilities then we are in trouble. Life is a powerful, dangerous thing. When we are considering creating new, potentially more powerful forms of life we need to consider how to do so without rendering ourselves extinct. The challenge is not to prove that we will not end up with a stable economically competitive non-combative equilibrium. The challenge is to look for every possibility that could lead to total war and make damn sure we&#039;ve considered them before we touch the on switch.

Carl&#039;s &#039;Reflective Disequilibria&#039; post is a good one. We cannot assume conflict with or between AIs will follow the trend of human wars. The competitive superiority of &#039;free men&#039; makes for some damn good movies and has a strong impact on human history. However, that is an ideosyncracy that a third generation em will probably not possess.
</description>
		<content:encoded><![CDATA[<p>There may be, as Robin notes, a bias towards seeing new tech through the lens of total war. There is also a bias for optimism, based off mellenia of shot messengers and a bias to see the future through the lens of the present.</p>
<p>I see in this post a warning against mentioning potentially disasterous consequences to AI. This scares me. When even rationalists ought to feel ashamed of considering such possibilities then we are in trouble. Life is a powerful, dangerous thing. When we are considering creating new, potentially more powerful forms of life we need to consider how to do so without rendering ourselves extinct. The challenge is not to prove that we will not end up with a stable economically competitive non-combative equilibrium. The challenge is to look for every possibility that could lead to total war and make damn sure we&#8217;ve considered them before we touch the on switch.</p>
<p>Carl&#8217;s &#8216;Reflective Disequilibria&#8217; post is a good one. We cannot assume conflict with or between AIs will follow the trend of human wars. The competitive superiority of &#8216;free men&#8217; makes for some damn good movies and has a strong impact on human history. However, that is an ideosyncracy that a third generation em will probably not possess.</p>
]]></content:encoded>
	</item>
	<item>
		<title>By: Hopefully Anonymous</title>
		<link>http://www.overcomingbias.com/2008/11/total-tech-wars.html#comment-391662</link>
		<dc:creator>Hopefully Anonymous</dc:creator>
		<pubDate>Sun, 30 Nov 2008 08:16:57 +0000</pubDate>
		<guid isPermaLink="false">http://prod.ob.trike.com.au/2008/11/total-tech-wars.html#comment-391662</guid>
		<description>Carl, it&#039;s fantastic that you&#039;re finally blogging. You should link to it in your posting name, like the rest of us.
</description>
		<content:encoded><![CDATA[<p>Carl, it&#8217;s fantastic that you&#8217;re finally blogging. You should link to it in your posting name, like the rest of us.</p>
]]></content:encoded>
	</item>
	<item>
		<title>By: Will Pearson</title>
		<link>http://www.overcomingbias.com/2008/11/total-tech-wars.html#comment-391661</link>
		<dc:creator>Will Pearson</dc:creator>
		<pubDate>Sun, 30 Nov 2008 07:54:18 +0000</pubDate>
		<guid isPermaLink="false">http://prod.ob.trike.com.au/2008/11/total-tech-wars.html#comment-391661</guid>
		<description>To increase your chances of survival you want to be have as little compute power as you can get away with. So that you don&#039;t waste resources on things that don&#039;t intrinsically get you more resources. Look at plants, fantastically successful, but no use for a brain and they would be worse off with one.

Greater computational ability doesn&#039;t automatically lead to winning. Only if there is something worth discovering or predicting with the intelligence, another energy source or a cheaper way of doing something you need to do to survive, does it make sense to invest in compute power.


</description>
		<content:encoded><![CDATA[<p>To increase your chances of survival you want to be have as little compute power as you can get away with. So that you don&#8217;t waste resources on things that don&#8217;t intrinsically get you more resources. Look at plants, fantastically successful, but no use for a brain and they would be worse off with one.</p>
<p>Greater computational ability doesn&#8217;t automatically lead to winning. Only if there is something worth discovering or predicting with the intelligence, another energy source or a cheaper way of doing something you need to do to survive, does it make sense to invest in compute power.</p>
]]></content:encoded>
	</item>
</channel>
</rss>

<!-- Performance optimized by W3 Total Cache. Learn more: http://www.w3-edge.com/wordpress-plugins/

Page Caching using disk (enhanced)
Database Caching using disk
Object Caching 429/446 objects using disk
Content Delivery Network via Amazon Web Services: S3: overcomingbias-assets.s3.amazonaws.com

Served from: www.overcomingbias.com @ 2012-02-11 19:08:24 -->
