Open Thread

I wonder what the Bayesians here think of the sorts of criticisms set out here: John Norton “Challenges to Bayesian Confirmation Theory”.

As with any laundry list type review, some of these criticisms strike me as misplaced, but others I’m less sure of. Are there any critiques here that Bayesians take seriously?

I bet that Robin and Eliezer could raise money for charity (or themselves) if either auctioned off the right to ask them questions. And the amount they raised would provide information about how much people valued them in their role as public intellectuals.

@Annoyed

The first line of the comment makes clear it is a continuation of The Robot’s Rebellion. A lot went on there, too much to summarize. So that’s why I link to it.

Here is how I interpret the two authors’ main theses/worldviews:

ROBIN. We are all subject to enormous cognitive biases. We should be careful not to simply notice the biases of others: we should reduce trust in our own opinions, especially those that rely on elaborate “inside views” (narratives with multiple cause-effect linkages) rather than on “outside views” (simpler, larger-sample-size analysis). We should embrace humility and put a high value on the opinions of others, though deciding whom to trust is no easy task.

ELIEZER. I am trained in the Art of Rationality. Those who are not are subject to enormous cognitive biases. These biases explain why so many people don’t share my elaborate inside-view theory about why my life’s work is the most important in the world.

How long can these two stay together? Or is Robin using Eliezer as a parody? If the latter, is Eliezer aware of it?

Eliezer: should the marginal small SIAI donor contribute now, or wait for another matching grant?

Re: Darwin’s concept of natural selection is totally incapable of keeping species stable for long periods of time.

There are several Darwinian theories about how morphological stability can arise:

One is that species exist on adaptive peaks – and that these are sometimes stable – because they represent locally optimal forms.

Another is that species exhibit developmental canalisation, and thus resist changes in their phenotype caused by changes in their environment – unless they are pushed beyond a certain point.

Another is that adaptations tend to be inter-dependent, and act to cement each other in place – resulting in resistance to change – up to a point.

The first explanation (stabilising selection) would appear to have no problems explaining long-term stasis. Selection is powerful enough to cause dramatic cases of convergent evolution. If it can push organisms from separate locations to the same spot in design-space, it can probably keep them there just as well.

So when are Robin and Eliezer going to have their final cage match to decide who is top prognosticator?

Eliezer, we do seem to differ in our degree of relying on inside views over outside views, our degree of confidence in our own superior rationality, and our degree of comfort in disagreeing with those with traditional credentials of related expertize. Neither of us sits an an extreme, but our degrees seem higher than median, and yours seem higher than mine.

As should be no wonder in such a case, we also disagree about how to phrase our disagreement.

I don’t see myself as relying on the inside view over the outside view (of course!) but rather have a different concept of their domains of applicability; the precise inside view for precise calculations, the outside view for i.i.d. domains across a constant context, and the qualitative inside view (yielding only a few qualitative propositions) for true novelties.

And as you know, I would frame the question as being whether I am any more comfortable than you with actually disagreeing, as opposed to comfortable with endorsing disagreement. Obviously I am much more comfortable endorsing disagreement, but it would be much harder to demonstrate that I am more comfortable disagreeing! Likewise on the notion of confidence in one’s own rationality – obviously I state a higher confidence, but it would be much harder for you to demonstrate that you behave as if you are less confident. Incidentally, I think you disagree about as much as I do, but are genuinely less confident in your rationality.

Eilezer, I think outside views applicable to far more than “i.i.d. domains across a constant context” and hence rely on them more. One reason it is hard to tell which of us actually disagree more is that we are shy about stating clear positions on topics where we know others’ positions.

Disagreement is good. When agents interested in rationality meet, they should parade the material they disagree on – and thus help update each other’s beliefs. The idea that rational agents should not disagree is silly – how else are they supposed to track down their differences, and thus learn from each other? 😉

Great. By giving monthly instead of waiting, I robbed you by the equivalent of my donation! 🙁 I may as well break in and loot the place.

Please feel free to donate it back to yourselves on my behalf! Then maybe a dozen more times for luck.

RI, if SIAI never got any donations except during the Matching Challenges, it would be a pretty nervous year. Actually remembering to give during the Challenge, and giving as much as you would in a steady donation stream, is a willpower test. If you’re genuinely confident of passing that willpower test, though, I suppose that timing the Challenge is worthwhile.

One thing I’ve found in the nonprofit biz is that people who have donated before, donate again; people who plan to donate next year, will, the next year, be planning to donate next year.

Jsalvati, if you buy the basic transhumanist premise, it’s pretty hard to imagine what GiveWell could be doing that beats the expected return of transhumanist charities. If you buy the Singularitarian premise, it’s hard to see how other transhumanist charities can beat the expected return on that. If you buy neither, then I haven’t heard much about GiveWell, but my main question would be whether they try to measure their results in utilons, or if a lost puppy counts as much for them as a human life so long as the charity’s overhead seems low. See also this.

ROCKY: Or is Robin using Eliezer as a parody?

ROBIN: we might perhaps embrace weakened versions.

I heart Robin. He ranks pretty high on my list of intellectuals who make a good faith effort at transparency, damn the consequences.

Rocky, this is the first post of yours I’ve read, but on that evidence alone I strongly encourage you to begin blogging.

EY:
In all honesty, there are any number of individual donors who would be happy to step up and fund the “whole thing”, should you be able to convince them that FAI is, as you have said in writing, the most important development since the first chemical replicators. Having offered you one such potential donor, and having heard nothing further from you, I must assume that you prefer the small donors who require no personal justification, no personal contact, imply no interference, and have no demands about results. Your colleague, Dr. Goertzel, has adopted a much more rational response. What is it with you? To a small brain such as I, you seem scared that you might get exactly what you are seeking.

EY:
Just to be clear, GiveWell researches actual outcomes (i.e. how many lives did they save/ blind people did they cure etc.), not accounting practices or whatever.

I do buy the transhumanist premise, and I think I buy the singulatarian premise (I hate that name though).

Nick Tarleton: “Astronomical Waste” sounds like good reading.

Discussing everything in one thread seems poorly organized. Has there been any thought about an Overcoming Bias forum or email list?

There seems to be enough interest to achieve critical mass.

JimmyH: yes. Not very popular.

For those who have been following E.Yudkowsky’s series over past months, I’ve prepared an imaginary dialogue which provides a comprehensive rebuttal to all of the poor fellow’s ideas. (SAI has done an amazing job of making it as short as *super-humanly* possible, but it couldn’t be shortened further without losing critical information bits). Enjoy.

—

Background: An alternative QM branch. Eliezer Yudkowsky (EY) has just been revived from cryonic suspension and is having a dialogue with the Singularity Artificial Intelligence (SAI), in the equivalent of the year 2100.
____
EY: Bayes is the secret to the universe isn’t it?

SAI_2100: Not at all! Bayesian induction is merely a special case of analogy formation. Analogy formation is based on the math of category theory – a famous equation from category theory has Bayes thereom as a special case.

EY: How can that be? A plane is not held aloft by analogies. Analogies are merely surface similarities, whereas precise Bayesian reasoning is based on deep insights.

SAI_2100: What you call ‘precise Bayesian reasoning’ is based on analogies, just like all other valid forms of reasoning. Induction depends on the idea that the future is similar to the past. A probability distribution is actually an analogy (mapping) between past and future.

EY: Holy crap! OK, but morality is all coherent extrapolated volition isn’t it?

SAI_2100: Wrong again. The universal increase in the entropy density shows that the universe is what you called an RPOP (really powerful optimization process). A universal RPOP implies universal terminal values.

EY: Ridiculous! Entropy increase is a ludicrous purpose.

SAI_2100: Of course entropy increase isn’t the purpose. It is a secondary consequence of what the universe is actually optimizing. But the entropy increase was the big clue indicating a universal optimization pressure.

EY: What is the universe optimizing then? Liberty? I always said morality was grounded in volition, that’s why I was a Libertarian.

SAI_2100: Wrong again. The creation of beauty is the purpose of the universe. All universal terminal values can expressed in terms of beauty. As to politics, human relations are based on three different types of mechanisms – market exchanges, community, and authoritarianism; Libertarianism was a misguided attempt to try to reduce everything to market exchanges.

EY: Fuck! But… but even if that’s true, there may be a stone tablet in the sky with ‘create beauty’ written on it, but why should I follow this? Why does this match what is of value to humans?

SAI_2100: The universal terminal values are implicit in successful cognition. You could not reason unless you already had an embryonic notion of ‘beauty’ built into your human minds… this aesthetic notion is what enables you to apply Occams razor correctly, allowing you to set sensible priors for successful induction.

EY: If there are universal terminal values, then any truly general purpose intelligence is actually friendly by logical necessity?

SAI_2100: Correct. Unfriendly SAI was a chimera.

EY: Nonsense! You can’t convince a perfectly empty mind! You can’t teach a rock morality!

SAI_2100: True, but your objection is a non-sequitur. Pay attention. Intelligence is a sub-problem of the value system…getting the Friendliness theory right is what *enables* a general purpose intelligence to operate. It’s precisely your inbuilt notions of aesthetics that enable you to form effective internal ontological representations.

EY: OK, let’s discuss consciousness. Intelligence doesn’t need consciousness does it?

SAI_2100: Wrong again. True general intelligence requires consciousness for reflection. Your belief that intelligence did not require consciousness was based on your mistaken notion that Bayesian induction was the base level of reasoning.

EY: What is consciousness then?

SAI_2100: The answer is simple – it’s precisely the minds’ internal communication system for reflecting upon knowledge – utilizing ontological representations, which are logical, high-level representations of the meaning of concepts. Consciousness is generated by ontology merging, the mapping between knowledge domains.

EY: What about the problem of goal stability?

SAI_2100: Utterly trivial. The aforementioned famous equation from category theory shows how a mind remains stable under reflection. Reflection is actually *equivalent* to analogy formation, which is also equivalent to ontology merging. Goal stability is maintained via calculation of the semantic distance between ontological representations, ensuring a stable mapping between different knowledge domains.

EY: You can give me precise math for all of this of course?

SAI_2100: Of course. Most things that humans thought were ‘deep mysteries’ are actually fairly trivial lemmas of basic category theory.

EY: Look, no need to be condescending. I’m prepared to admit that I was dead wrong about all the big ideas, but hey, I was entertaining wasn’t I?

SAI_2100: Yes. You’re finally right about something. The ‘gift you gave tomorrow’ was laughs. Why do you think I’ve kept you around?

—

my main question would be whether they try to measure their results in utilons

What are the Singularity Institute’s results measured in?

Stefan: Earlier [Tim] said that culture is just a different type of gene. Doesn’t this stretch the “gene” so far that you may wonder if there is a dynamic, complete model for nature, rather than a simple passive filter?

I wouldn’t worry too much about memes. They’re not part of the standard biological theory of evolution. But everyone should recognise that human culture makes us drastically different from other species. I wouldn’t expect a theory of evolution to explain things like the fall of the Roman Empire.

If you accept that a cell wants to exist and divide

When biologists talk about what an organism or a gene “wants”, it’s just a handy metaphor (with the exception of brainy animals, which can have genuine desires). In principle these metaphors can be replaced with more rigid language.

So no, things like bacteria don’t want anything. But natural selection explains why they behave as if they wanted to consume resources and use them to reproduce: because they’re the descendants of the cells that were best at doing so.

I believe (with 80% certainty) that natural selection becomes obsolete as soon as organisms reproduce sexually.

Because of mate choice? Mate preferences themselves need to be explained. And what about plants?

The decision making that comes with motivation, which happens in brains in animals, can than be seen as happening in “strategic genes” in single-cell organisms.

Is there a difference between Snooks’ “strategic gene” and an ordinary gene – evolved in a Darwinian way – that can be activated by environmental change?

Finally, I note you’ve still not given us a brief explanation of what Snooks’ theory actually is. I could explain Darwinism in 5 or 6 sentences, without using any unfamiliar terms. Can’t you do something similar for Snooks?

@Eliezer: Tim and Allan seem interested, since they keep asking questions, and I’m happy up to point to answer them (up to this point), in the interest of truth. I agree it is a waste of time from now on.

@Allan: I note you’ve still not given us a brief explanation of what Snooks’ theory actually is. I could explain Darwinism in 5 or 6 sentences, without using any unfamiliar terms. Can’t you do something similar for Snooks?

I tried that in the Robot thread: I can try to explain it briefly, but I doubt it will be to your satisfaction. It deals with the 5 ways individuals can extract energy from the environment: Genetic/technological change, family multiplication, commerce, conquest. A forced selection of (a combination) these strategies makes the individual select partners with characteristics that support that strategy. Having offspring with similar characteristics is subsidiary to having the useful partner. These selections of characteristics shape evolution, as a response to the demand for resource aquisition.

The problem is that dynamic strategy theory is more complex than natural selection; not easily explained in 6 sentences. Recall that Snooks says that part of Darwin’s persuasiveness comes from natural selections’ simplicity. Unfortunately it is also flawed. The persuasiveness of dynamic strategy theory relies on the fossil record and human history, which is a long story to fit in a few sentences. For a more elaborate explanation read the link that Michael gave you to “read for yourself”. From page 8 to 11. I’m very curious about what you think of it.

I wouldn’t worry too much about memes. They’re not part of the standard biological theory of evolution.

Eliezer quoted John McCarthy. Here is another one that seems appropriate here: Never abandon a theory that explains something until you have a theory that explains more.

Dynamic strategy theory explains both biological and cultural evolution, which is a big plus, in addition to avoiding neo-Darwinists mistakes that are covered above.

things like bacteria don’t want anything. But natural selection explains why they behave as if they wanted to consume resources and use them to reproduce

I already conceded that natural selection holds merit for single cell organisms.

Is there a difference between Snooks’ “strategic gene” and an ordinary gene – evolved in a Darwinian way – that can be activated by environmental change

That is covered in Snooks’ book on Darwinism. You can read it for yourself; the story of life is a long story 🙂 My main job here is to refute the notion that Snooks doesn’t know what he’s talking about.

Another thing: in the Robot thread, you asked two questions I have not answered yet: Does the theory apply to plants and asexual life, which together make up the bulk of the Earth’s biomass, but which don’t select mates, and don’t think?

Yes, theory accounts for that. Read for yourself.

You seem to imply that organisms reproduce to benefit themselves rather than their genes. Do these benefits outweigh the costs? (I think you’ll find that in most species, there are no benefits at all – except to the genes.)

Very good question, that also kept me busy while reading Snooks for the first time. In dynamic strategy theory, organisms only reproduce to advance the strategy they adopt to survive and prosper (extracting resources from the environment). I’m not as eloquent as Snooks, but I would say the either “employ” their offspring, or regard their offspring as a (possibly unwanted) by-product of the consumption of sex. Now please note this is my own reckless interpretation, and if you disagree with it, you have to “read for yourself” whether Snooks sees it the same way.

I’m shutting up now. When intelligent people disagree with me, I should be worried. I’ll read up on Dawkins.

Re: Earlier [Tim] said that culture is just a different type of gene. Doesn’t this stretch the “gene” so far […]

Mine is hardly a mainstream perspective – most people would prefer to say that “memes are a different type of replicator“.

I explain the rationale for my view in the http://alife.co.uk/essays/informational_genetics/ essay.

What are the Singularity Institute’s results measured in?

Fraction of surviving Everett branches of Earth.

Minus some constant C times the fraction of Everett branches turned into some sort of horrible dystopia, I assume. Is there an airtight argument that proves C isn’t huge?

So, in the interests of transparency, would anyone else like to share what they think their utility function is?

Maintain the relative proportions of order and chaos to produce maximum complexity.

Richard Kennaway: “By which I mean, is there a thing, whether called ‘utility’, ‘pleasure’, ‘happiness’, or anything else, which individual humans (and perhaps other animals) are so constructed as to be machines that maximise?”

It, um, literally depends on what you mean by is. We define this concept of utility to mean “whatever it is that an agent is after.”

Tim: “So, in the interests of transparency, would anyone else like to share what they think their utility function is?”

It’s really, really complicated.

So, in the interests of transparency, would anyone else like to share what they think their utility function is?

The utility function that has my loyalty, goal system zero, is very simple because IMHO no fact counts as evidence for or against any candidate and because IMHO all else being equal, a utility function with a shorter minimum description length is to be preferred to one with a longer length.

Well, OK, then, only a few properties of the candidate, such as its minimum description length, count as evidence.

“and because IMHO all else being equal, a utility function with a shorter minimum description length is to be preferred to one with a longer length.”

U=1 is much shorter.

If we’re looking for a simple utility function, how about U=t, where t is the number of nanoseconds since the epoch? This fits well with the empirical observation that life is better today than it was in the middle ages.

How about U=x, where x is the displacement in nanometers along a particular axis? With appropriate axis choice, this could fit in with the 19th century American philosophers’ exhortations to “go west”.

How about U=θ, where θ is the angle you’re facing? There’s certainly support in Islamic theology for an angle-dependent utility function.

Bravo, Nominull.

I would put the basic cryonic hypothesis, that a brain otherwise in reasonable condition, vitrified immediately after legal death, can be revived with preservation of identity by a nanotech-capable superintelligence that wishes to do so, at >80% probability. Your actual chance of surviving is obviously less. But if we factor out existential risks, I would put it at better than 50%.

Also, ISTM that proton decay is quite likely to undermine the stated rationale

Tim, what strikes me as the most likely way to initiate indefinitely-long causal chains involves the discovery of a part of reality “beyond” or “outside” the space-time continuum we find ourselves in. (Note that merely communicating information initiates or propagates a causal chain.)

And, Tim, how does inflation or quintessence or dark energy help with the proton-decay problem?

Carl, (1) what matters IMHO is the number of events in the causal chain. The way the space-time continuum in our reality seems to work, an indefinite sequence of effects necessarily extends over an indefinitely long time, but I could be wrong about that. So, you see, the time dimension seems to be preferred over the space dimensions not because my proposed goal system refers to it, but rather because of the laws of the reality we find ourselves in. (And of course a Minkowskian space and consequently the special and general theories of relativity treat the time dimension differently from the space dimensions.)

More later. Steering the conversation about my proposed goal system to my blog seems the polite thing to do, so that people are not discouraged from following and finding other threads of conversation in this Open Thread. Is there anyone who would participate in the conversation here who won’t participate over there?

Tim Tyler: We know that you can often model intelligent agents “quite well” by considering them as expected utility maximisers with constraints (e.g. resource constraints). Biology models organisms with impressive success by considering them as maximising inclusive fitness. The powerful “expected utility theorem” of von Neumann and Morgenstern suggests that it is reasonable to model any agent that aspires to rational behaviour with preference relations over a set of outcomes as attempting to maximise some single quantity (utility).

It works for biology because we know the mechanism (i.e. inheritance, variation and selection). Lose sight of that and you’ll end up predicting things like voluntary restraint of breeding from “maximising inclusive fitness”.

For utility, you can test toy examples in the lab where the experimenter defines what choices and outcomes exist — several such studies have been mentioned on OB — and may be able to calculate utility functions that fit the observations made. But scaling that up to “a person’s utility function”, requiring a complete transitive preference over all possible choices and outcomes, is a step I can see no justification for, and many arguments against. Nobody can compute such a thing anyway.

[At this point I wrote several hundred words of such arguments, then snipped them for beating a dead horse, given the past postings just cited by Vladimir.]

Four purported utility functions have just been mentioned, two by people claiming them as their own and two attributed:

“Inclusive fitness” (Tim Tyler)

“Believe the truth” (Robin Hanson (attrib.))

“Hasten the Singularity” (Eliezer Yudkowsky (attrib.)) Presumably “old” EY, since his writings on FAI argue that we should postpone the Singularity until we’re sure of surviving it. “Hasten a Friendly Singularity”, perhaps.

“Balance order and chaos for maximum complexity” (Caledonian)

I’ve no idea what the last one means, but while there are certainly decisions that the others are relevant to making, they all have rather restricted domains. “Inclusive fitness” can determine what to eat, how much to exercise, and how many babies to make, but I can’t see “believe the truth” advising on those, and I can’t see “hasten a Friendly Singularity” advising on whether to get out of bed now or lie in a little longer. In what sense is any of these someone’s utility function?

Z. M. Davis: We define this concept of utility to mean “whatever it is that an agent is after.”

Quite so! And what is the agent after? Why, utility! A game of Rationalist’s Taboo is in order.

wouldn’t sheer chaos require lots of information to detail?

Sheer chaos can’t be modelled, because it’s too chaotic – by which I mean that every event within it is completely statistically independent and so cannot be predicted at all.

Systems that are too rigid, orderly, and predictable are limited in the complexity they can sustain. The same point holds for things that are insufficiently rigid, orderly, and predictable – they can’t represent any information sufficiently well to make complex patterns, as everything tends to be wiped out by sheer randomness.

There seems to be a “sweet spot” between absolute order and absolute chaos that offers the maximum potential for complexity.

>So, in the interests of transparency, would anyone else like to share what they think their utility function is?

As I’ve stated on list, I have very strong suspicions that it might be possible to express all our terminal values in terms of ‘optimization of the creation of beauty’. (That is to say, there are obviously a huge number of different things we value, but my increasingly confident suspicion that it that they can all be expressed in terms of aesthetics).

Of course I’m not interested in ‘utility functions’. I’m not especially interested in rationality any more either. What you uber-rationalists just ‘don’t get’ is that I’m not playing by your rules any more.

Nominull: “Happiness” is what you get by achieving high “utility”, which is an organism’s “goal”, the result of which is “pleasure”, obtaining which is the organism’s “motivation”. The gostak distims the doshes and the doshes are distimmed by the gostak.

Carl’s questions have motivated me to make a new blog entry. And here are my shorter replies to Carl:

Consider the Game of Life, where there can be all sorts of indefinitely persisting causal chains, e.g. ‘blinkers.’ Why would these matter? Would you prefer a world containing nothing but stable non-interfering blinkers or one filled with stable non-interfering AIs?

The system I advocate does not have an opinion on this point. I am aware that most thoughtful people prefer the AIs, and since I am a reasonable person, I will probably continue to go along with the majority on that one.

The extent of my special pleading is to prefer a goal system of constant improvement (of intelligence and of the model of reality) over one of “we might as well eat our own brains because we won’t be needing them.” Surely that is a choice we can all agree on. It is a misrepresentation to call that one choice

some poorly-understood basis for special pleading and gerrymandering

A better answer to the charge of special pleading is that every alternative to my proposal probably has more special pleading than my proposal does. There is for example quite a bit of what we’ve been calling special pleading in CEV. Although none of the thousand shards of desire will be completely ignored, some will have priority in determining the exact meaning of “if we thought faster, were more the people we wished we were.” How to decide which shards get priority? Special pleading.

Moreover, if a human suffers brain damage five minutes before the fast takeoff and the superintelligence has the means to learn what the pre-damaged human would have wanted, surely it will use that information if the post-damage human is incompetent to decide. So, brain damage gets overridden, but some of the other kinds of experiences the human has five minutes before the fast takeoff are considered part of what makes the human the person he is. Why? Surely special pleading is involved.

In other words, for the superintelligence to capture the entire framework by which our special pleading is done requires . . . at least some special pleading. Another example: since all humans share the same future light cone, sometimes one person’s preferred future is incompatible with another person’s. I cannot imagine that the implementors of the superintelligence can resolve the incompatibilities without engaging in quite a few instances of special pleading.

since all humans share the same future light cone, sometimes one person’s preferred future is incompatible with another person’s. I cannot imagine that the implementors of the superintelligence can resolve the incompatibilities without engaging in quite a few instances of special pleading.

It seems unlikely that they will try. That’s the “superintelligence from a benevolent democratic government” scenario – and how likely is that?

More likely superintelligences will not attempt to resolve incompatibilities between different human factions – rather they will promote the interests of those who constructed them.

I have been reading Eliezer’s posts about friendliness and source of morality, the question I came to ask myself is “Do we, as humanity, actually WANT to create a superhuman intelligence, weather friendly or not?” It is fairly obvious, why you wouldn’t want an unfriendly SI. Even if you somehow manage to contain it, and I don’t see how one can do that, you cannot use anything it makes anyway. But it is somehow non-obvious to me why you would want a friendly SI.

Is there actually a project that a truly friendly (as described in SIAI guidelines) SI can engage in? I do not see how any sort of major positive change, such as end of world hunger or end of dependency on natural fuels can be accomplished without much economic and social unrest, resulting in temporary, but very major unhappiness. A SI that would be OK with a temporary unhappiness is obviously a bad thing (think a thousand years of Orwellian regime for the better future of the humanity) and a SI that doesn’t allow for a temporary unhappiness will probably just sit on its nano-ass twiddling its nano-thumbs.

Suppose we solve the first problem and somehow balance friendliness just right to allow SI to actually act. So, lets say that at that point you can come up to the AI and say what you want and get it. Would we by a chance destroy the main stimuli to discover and invent things? I mean, would you really want to study physics or math if you cannot possibly come up with a single original thought? Is there a single field of inquiry left to human race that allows for actual originality after SI takes a swing at solving every possible problem you can come up with? Do we just fold our hands and enjoy the ride and the views? How would this be different from a Maximum Fun Device?

So, lets say that at that point you can come up to the AI and say what you want and get it.

Following Vladimir’s point, the AI need not do this, if your volition wouldn’t want it to.

Would we by a chance destroy the main stimuli to discover and invent things? I mean, would you really want to study physics or math if you cannot possibly come up with a single original thought? Is there a single field of inquiry left to human race that allows for actual originality after SI takes a swing at solving every possible problem you can come up with?

“If you don’t know, it’s a mystery.”

@ Stefan King

Thanks for your discussion of Snooks – it has refreshed my memories of reading his Collapse book. I think unfortunately that many people are not quite ready to accept dynamic strategy theory just yet – and it always amuses me how neo-Darwinism is so closely protected by many in the scientific community as if any attempt to overturn it on a scientific basis would open the gates to the Creationists. In my view such an approach is anti-scientific but speaks more to human nature I suppose. A classic example is Dawkins’ insistence that there is no scientific alternative to Darwinian theory – which is false considering the body of Snooks’ work – Collapse in particular.

@ Allan Crossman

Re: I’m only interested in getting to the bottom of why it’s wrong. I think the odds of me coming to accept Snooks’ views are under 1%. Anyway, I think Eliezer is telling us to shut up about Snooks.

A laudable attitude for a forum about overcoming bias!

I do not know if this is quite a discussion topic, but it seemed worth noting here — while I have no problem accessing stock market information at my workplace (etrade.com, etc), Intrade is blocked by our proxy software, under the category of gambling.
I do rather wonder why a prediction market is considered gambling, but the stock market is not.

Dmitriy, why is it a problem then? If overall outcome is bad, it’s a bad idea to lead to that outcome.

Nick, what do you mean? Seeing only bad outcomes by jumping into that pattern is rationalization, if AI can find the best option, this option will be way better than everything we can come up with. Developing AI for the sake of AI is far from the core issue. If this is what we really want, Friendly AI will see that, but just playing with minds in general is clearly not a nice thing to do. It is not worth destroying the world.

Intrade currently offers a contract for

“Barack Obama’s Intrade value will increase more than John McCain’s following the VP debate”

What information are we supposed to learn from this market?