28 Comments

"The first derivative of s will be equal to he, e will be a function of s, and h will be treated as fixed in the short run. In order for growth to proceed with a steady doubling, we will need e to be a very specific function of s, and we will need a different function for each possible value of h. Reduce it much, and the self-improvement will slow to a crawl. Increase h by an order of magnitude over that and you get an immediate explosion of improvement in software, the likely aim of a leader in emulation development."

I'm not exactly sure how to interpret this.  Could someone who thinks they understand explain using equations?

Expand full comment

Michael, James (a professional economist) is right here; you are wrong. Professionals know things!

Expand full comment

michael vassar - But if Firm B can raise as much as it wants from capital markets (and do so very quickly) it wouldn't get outbid for input resources.

Expand full comment

James Miller: The owners of firm A have massive wealth in the short term, as does firm A as an entity. Firm A can raise more money on capital markets thatn Firm B can. Both firm A and firm B can use the same limited supply of people with the relevant skills and of computers to improve their products. Firm A can thus out-bid Firm B for the resources which Firm B needs to improve its design.The key point is that car companies can't do this but upload companies can because the market for cars early in the history of the automobile is limited compared to the global supply for steel, rubber, etc. By contrast, the market for high quality mental labor at a low price is huge compared to that for at least some inputs into R&D or into computer production so those inputs can be exhausted completely.

Expand full comment

'Say what? Rebellion? I'm turning these bastards off!'

Or detonate our secret EM pulse bomb.

Expand full comment

Good to see more by Carl.

Expand full comment

Hmm

What's the opposit of a neutron bomb?

Anyway, we could always put a switch on the robots...'Say what? Rebellion? I'm turning these bastards off!'

Expand full comment

"But if firm B is expected to earn a long term positive economic profit it could raise all the money it wanted on capital markets."

Provided that contract enforcement and property rights are secure, so that lenders believe they will be repaid, and can be approached without resulting in government expropriation. The expropriation concern is why my discussion above focuses on ways to acquire hardware/funds without drawing hostile attention. However, I did mention lending, as "promising the Moon," since while a firm using loan funding to conduct an in-house intelligence explosion could promise absurdly high interest rates, if it were successful creditors would no longer be able to enforce a contractual obligation for repayment through the legal system, and would need to rely on the honor of the debtor.

Expand full comment

Carl wrote in a comment "Initially sole knowledge of cost-effective em design means that you get a vastly, vastly, higher return on investment on research expenditures than others do."

Let's say that firm A has the cost-effective em design whereas firm B has a cost-ineffective em design. Imagine that it will take firm B lots of time and capital to develop a cost-effective em design.

True, give both firm A and firm B a dollar and firm A could use it to generate more revenue than firm B could.

But if firm B is expected to earn a long term positive economic profit it could raise all the money it wanted on capital markets. There would be no financial constrain on firm B and thus no financial market advantage to firm A even if firm A could always earn greater accounting profits than firm B.

(Economists define profit taking into account opportunity costs. So let's say I can do X or Y but not both. If X would give me $20 and Y $22 then my economic profit from doing Y is $2. In contrast an accountant would say that doing Y gives you a profit of $22. I'm not assuming that Carl doesn't know this.)

Expand full comment

Carl, I can't win a word war of attrition with you, where each response of size X gets a reply of size NX, until the person who wrote the most crows that most of his points never got a response. I challenge you to write a clear concise summary of your key argument and we'll post it here on OB, and I'll respond to that.

Expand full comment

Eliezer, the opportunity cost of any product is the revenue you would get by selling/renting it to others, not your cost of producing it. If there were a big competitive advantage from buying wholesale over retail from yourself, then firms would want to join large cooperatives where they all buy wholesale from each other, to their mutual advantage. But in fact conglomerates typically suffer from inefficient and inflexible internal pricing contracts; without other big economies of scope conglomerates are usually more efficient if broken into smaller firms.

Expand full comment

Many, many information companies choose to keep their source code private and sell services or products, rather than selling the source code itself to get immediate wealth.

Expand full comment

"Many, many information companies choose to keep their source code private ..."

That's true and many give pretty complex software away for free. It would seem to me the more general\platformy\frameworky a tool is, the more likely it is to be sold as a product by itself. If there are 10,000 different business models that can use an ems then your opportunity cost is related to the difference in time it would take for you to spin up those businesses yourself vs. letting established businesses run in parallel and getting a piece right off the bat. Though I would confess the implied serialization of the process in the former case can be blamed quite extensively on limited brainpower (as can most things).

What is the best current example of withheld software that would be easily monetized for riches if sold directly?

Expand full comment

Carl, thanks for your reply.

"I mentioned the increase in mean ability within the workforce, and agree that talk of (elite, in this context) human-equivalents is a crude modeling technique that is only applicable to the precursors and perhaps earliest stages of an intelligence explosion."

I guess I am attempting to do an end-run on your reasoning by suggesting that attempted computer modelling of non-elites would be somewhat insane if you had even a snowball's chance at the elites. Why build Joe the Plumber if there's any chance you can go directly to building Einstein at similar cost?

It seems bizarre to my mind to imagine AI as an economic good, and think of the technology justifying itself economically or funding its own development.

It's quite clear the human race knows perfectly well that 'the first decent AI wins the world'. Look at how much money military projects have already invested this century into no-hoper symbolic AI that could barely push blocks around a table. When America realised 'we can put a man on the moon', it threw huge amounts of GDP at the task. Building the second Einstein - if it looks like there is anything more than a snowball's chance in hell of it succeeding - is highly likely to attract huge, almost unlimited funding.

If you could do a demo right now showing an AI brain simulation that is genuinely as smart as a cat, I can almost guarantee you would get a vast budget from the military, a personal army of researchers.

So, without meaning to insult you, I feel an economic modelling of the envisaged 'early days of working brain simulation' is about as useful as a similar economic modelling of 'putting a man on the moon' would have been, 100 years ago. Once we can simulate a cat, reliably and usefully, I believe whoever does that simulation gets the budget from the military to take it all the way to Einstein.

"On your nuclear point: James Hughes has suggested that such use of nuclear deterrence be an integral enforcement mechanism for a treaty regulating the development of artificial intelligence."

Interesting to know - thanks! :-)

Anonymous.

Expand full comment

Many, many information companies choose to keep their source code private and sell services or products, rather than selling the source code itself to get immediate wealth.

Expand full comment

"I'm not saying that the leading em-builders are getting ems from nowhere without paying opportunity costs"

The opportunity cost comes from not selling the very valuable thing in your possession, thus depriving you of assets you could use for generating more wealth immediately.

Though it's really all a gamble. Maybe your long-term profitability goes up by delaying the sale of your tech, or maybe it doesn't. There are no guarantees. Perhaps there is a competitor closer to you than you thought, or is moving faster now for whatever reason. Perhaps the company that DOES sell now finds a way to use its fabulous new found wealth to catch up to you. Maybe the first mover manages to maintain customers even when you deploy a superior product for non-technology reasons. The closest thing to a guarantee is selling now for the cash.

Expand full comment