Open Thread

The Harlem Renaissance section here is a striking example of raising your goals enough to make success possible rather than just putting out the usual amount of effort.

I’ve written an essay about the wirehead problem – check it out if you are interested: http://alife.co.uk/essays/the_wirehead_problem/

The essence of wireheading is the bypassing of evaluative functions and producing reward sensations directly.

A paperclip-manufacturing AI wouldn’t be wireheaded by definition, because it has to look at the world and detect certain configurations in order to feel rewarded. A wireheaded AI wouldn’t care about external conditions at all — it would just feel great, all the time, for no reason, and regardless of whatever else was happening.

A wireheaded AI wouldn’t care about external conditions at all – it would just feel great, all the time, for no reason, and regardless of whatever else was happening.

Maybe – though the example of the heroin addict suggests things are not necessarily always that simple. Curt once gave Enron as an example of a company that had stopped trying to make shareholder profits (the normal utility function for such a company). Yet it still acted as though it had some preferences – it behaved as though it wanted to cover up its accounting scam for as long as possible.

I haven’t tried to formally define wireheading in the essay – but it doesn’t seem critical to understanding the basic problem. Essentially, wireheading is a change resulting in the generation of reward for something that “shouldn’t” generate reward – to the point where decidedly odd behaviour results. (Interpret “shouldn’t” as you will.) There’s also a kind of “negative” wireheading based on pain-killing.

Hi. In 3 weeks there’ll be even more Bayesians in the Bay Area than usual. Would anyone, locals or visitors, potentially be interested in an informal OB meetup near to that weekend, possibly on an evening or the Sunday?

One of the things I have scheduled – it remains to be seen if I’ll get there, because I’m already on overtime – is a sequence on Fun Theory. That answers the objection with respect to the future of humanity. In current practice, the answer is that it isn’t necessarily true that you can get more scientific work done without an SO; that will vary depending on temperament, resources, and of course the girlfriend in question. The calculation is worth doing, but it’s not a foregone answer one way or the other, and I have no intention of going into the details in my case.

A paperclip-manufacturing AI wouldn’t be wireheaded by definition, because it has to look at the world and detect certain configurations in order to feel rewarded. A wireheaded AI wouldn’t care about external conditions at all — it would just feel great, all the time, for no reason, and regardless of whatever else was happening.

This relies on being able to distinguish internal and external worlds. If the paperclipper is so powerful that you might as well call the solar system its “body”, how is detecting configurations in the nearby physical world different from detecting impulses in your brain?

Why are such convictions, the denial of the primacy of human values, needs, and instincts, disagreeable?

Because there’s no light in the sky, outside of humanity, for values to come from. If you reject all our evolved preferences as philosophically invalid, what’s left?

Would anyone, locals or visitors, potentially be interested in an informal OB meetup near to that weekend, possibly on an evening or the Sunday?

Yes. (Visitor, don’t know yet exactly which days.)

Thought for the day–probability theory and decision theory push us in different directions: induction insists that you cannot forget your past; the sunk cost fallacy demands that you must.

Recovering: “Would anyone […] be interested in an informal OB meetup near [Oct. 25]?”

Count me in!

anti: no. the notion, as I understood it, amounted to this:

When we say “should/moral/etc…” we mean something. We may not fully be able to articulate that meaning, and we may have trouble working out what actually fulfills the various criteria corresponding to that, but to the extent that there is an associated meaning/question/computation encoded into our brains that’s associated with the relevant words, that’s what we ought to appeal to.

That does _not_ mean it’s “oh, whatever people happen to value”

it’s more the notion that the term morality refers to something specific. It happens to be that people tend to value this stuff called morality. And the beings that don’t value it, well… they’re by definition immoral, so there’s a limit to how much their opinion ‘should’ count. (should, of course, being a word that translates to whatever those partly “black box” criteria of morality ultimately turn out to be)

To the extent one rejects this notion, one’s going to have trouble talking about morality at all. I mean, presumably you mean something by that. Even if you can’t articulate precisely what you mean, or what the outcome of the “morality computation” is is something you can’t at this time determine accurately. To the extent that it does mean something specific (That is, that it’s a lever in your mind to a certain “black box” that computes morality), it doesn’t matter what people think is moral.

(did that come out relatively clearly)

ie, recall the distinction between a calculator trying to calculate the answer to the question “what’s 3 + 5” and a calculator trying to calculate the answer to the question “what does this calculator think is the correct answer to 3+5”? The latter can be more or less anything, but the former has a unique answer.

Regarding: Past behavior, the one you used to know. 99

Please don’t tell me this moment is not a bias:)

http://ca.youtube.com/watch?v=8OyD_ZfqXXw

Anna:) My abstract view

I have no immediate peers. Do a lot of OB readers have my same problem? I still stand by my decision to stay out of college, but I wish I could be around people with similar interests. I do have friends in college, but none of them are as passionate as I am with my interests. Math, writing, reading, good movies and music…. Yet, I feel like I have many years to go before I can contribute anything to any of these fields (meaningful or not, I feel obligated to try).

This is of a few places that I can go and feel among “my people.” How many of you are like this?

Thomas Ryan: “[…] How many of you are like this?”

At least one! I’m a lonely dropout-cum-generalist-autodidact as well. You can email me at: zack m davis {-at-} yahoo point cahm (no spaces; you will forgive these cumbersome antispam measures) if you want to talk.

I’m sorry readers have had to endure another month of straw-men, misconceptions, non sequiturs, ideology and superficial analysis.

—

In particular, the idea that intelligence is somehow reducible to a purely functional description (‘Bayesian Induction’, ‘Optimization’) could be a *big* mistake. That’s *one* aspect of intelligence – *optimization* is a big insight to be sure, but I don’t for one moment believe that that’s sufficient to encompass a full definition of intelligence. A more abstract (higher-level) description would base intelligence on *the aesthetics/elegence/simplicity of ontological representations*. I don’t for one moment believe that calculation of semantic similarites (the basic operation at the ontological level of abstraction) is reducible to Bayesian Induction, although of course there would have to be a Bayesian component to it.

What if it turns out that Bayesian induction is not sufficiently general to fully encompass intelligence? In summary, knock over the ‘Bayesian Induction’ domino, and the rest of the AGI stuff posted here woud collapses like a house of cards, you EY fan-boys realize?

—

Libertarian fan-boy faith has also come crashing down, with the US economy in near total melt-down. This was a common feature of ‘free’ markets as far back as records go. Thanks goodness the Libertarain ideology prompted by many self-proclaimed *geniuses* here will never be implemented.

The moral of all this is this great quote:

Conservatism is suspicious of thinking, because thinking on the whole leads to wrong conclusions, unless you think very, very hard

-Roger Scruton

Thomas Ryan, Z. M. Davis:

I would think there are a good number of OB readers like this. Myself included, and although I was raised homeschooled and I am seriously considering college I find autodidacticism pretty appealing. [My email is naxicasa {-at-} gmail point cahm, if you care.]

Indeed, I’m a wirehead and I like it!

🙂

Eliezer, are you implying there is some goodness that can’t be simulated?

Did I interpret that wrong?

In the movie Pi, whenever the protagonist gets stuck, he restates his premises:

1. Mathematics is the language of nature
2. Everything around us can be represented and understood through numbers
3. If you graph the numbers of any system, patterns emerge
4. Therefore, there are patterns everywhere in nature
Hypothesis: Within the stock market there are patterns as well.

If that movie was about an AI researcher, what would his premises be?

@Ian

“In the movie Pi, whenever the protagonist gets stuck”

Ian C., may I point out that Max, the protagonist in Pi, is crazy? He is paranoid, delusional, self-harming, and as Darren has stated, “addicted” to a self-created monster. I wouldn’t look to Max as a reliable guide to any sane endeavor. Unless you are truly interested in the premises of a lunatic AI researcher?

Max wasn’t crazy. He was afflicted with chronic migranes, possibly resulting from his intuitive understanding of the mathematics behind a very strange attractor that may be involved with the spontaneous generation of life.

Unless your premise is that the entirety of the movie was nothing more than a series of hallucinations, Max was perfectly sane. Impressive, given that he was pursued by both Wall Street executives and hysterical Kabbalists.

Eliezer, are you implying there is some goodness that can’t be simulated?

Another holodeckist (holodecker?? holodeckard??). Gosh, there are lots of them!

I’m impressed by how much compartmentalization the brain can do. For example, I have a quite different viewpoint on politics, society, and the world when discussing these things with my roommate than I do back at home. Yet, both viewpoints seem to be built of very strong and genuine convictions. It seems we humans will vary much of our thought depending on the group we are around.

I’d be interested to hear more analysis of mating and reproduction, though I don’t have specific to contribute at this point.

@Roger Scruton

Please, avoid inflammatory rhetoric that doesn’t contribute. Save that for Youtube commentary. Thanks.

About the ig Nobel winners: I think that it is quite amazing that slime molds are capable of learning.

Ig Nobel:

PHYSICS PRIZE. Dorian Raymer of the Ocean Observatories Initiative at Scripps Institution of Oceanography, USA, and Douglas Smith of the University of California, San Diego, USA, for proving mathematically that heaps of string or hair or almost anything else will inevitably tangle themselves up in knots.
REFERENCE: “Spontaneous Knotting of an Agitated String,” Dorian M. Raymer and Douglas E. Smith, Proceedings of the National Academy of Sciences, vol. 104, no. 42, October 16, 2007, pp. 16432-7.

If only one could extract energy, or any utility, out of knotting,

The prevention of death for all humanity has not grabbed me intellectually, like it seems to have done for other thinkers.

IMO, the most probable way that the “death” problem will be fixed is to use machines whose brains can be backed-up and copied. Since I doubt there will be very many uploads, full implementation of that solution will most likely entail the eventual deaths of most humans.

I have a normal aversion to self-harm, but no fear of death. I also have no fear of public speaking at all.

This also describes me… I don’t know if I have an attachment disorder as such, but my mother says that I lack empathy for other people and care too little about how other people view me.

I think saying “our goals are more complicated than just happiness” is a refusal to consider the question of what our goals are.

There are other reward states besides happiness. Much of human satisfaction lies in switching between various reward states — by themselves, any one of them quickly becomes stale and uninteresting.

“parents tend to be considerably less happy than the childless”

On self-reported happiness scales.

Douglas, your recounting of the research is probably more accurate than mine. But I do think that it goes slightly beyond “time spent with children”, rather to “years lived with children”. People with grown children are happier than childless people of the same age, but people still raising children are less happy though more fulfilled. Does that sound about right? (Admittedly I’m being lazy in not looking this up again. So goes the internets.)

I would like to see some discussion of the housing bubble and bailout plans. Specifically, the times when the government intervenes into the market price mechanism; this time its saying a bunch of mortgage-backed securities are undervalued by the market.

Typically we rely on markets to set prices, knowing they do so better than any other mechanism. Occasionally this mechanism seems to “break”. However, does that mean its rational to switch to another (political?) mechanism? Do we have a system that can accurately predict when markets go awry? If we do, do we have something that can out-perform distorted markets? Or is all of this just a “do something!” bias?

pdf23ds

Do you think we should optimize for What-People-Want instead of happiness/pleasure? That seems like a viable alternative to What-Makes-People-Happy, but I don’t think I understand it. Let me think “out loud” here.

There are some cases where people want things that are bad, because they’re wrong about something. Like, what if I wanted to stab myself because I thought it would feel great? Also, what happens when two people want mutually exclusive things? You have to measure the things against each other, and it seems like the way to decide which thing to do is to pick whichever one brings the most happiness.

As for parents bringing up children, my understanding is that it might make the parents less happy but the good upbringing makes the children much more happy, for the rest of their lives. This still seems like it’s optimizing for net happiness.

So, the question is, are we optimizing for something other than happiness?

According to most biologists, yes. As with most other organisms, the utility function for humans is well modelled as “expected number of grandchildren” – with the “expectation” being based on the assumption that we are in something like our ancestral environment.

Is this a utility function a good match for hapiness? Probably not, at least according to this: “second and third children don’t add to parents’ happiness at all. In fact, these additional children seem to make mothers less happy than mothers with only one child”.

I would say that it’s bad to become an opiate addict because negative feedback mechanisms within the brain limit the effectiveness of opiates to produce sustainable pleasure. In other words, you eventually lose your capability to experience pleasure from both opiates and the events that naturally trigger that particular reward system. In the long run, you end up less happy than if you had never started taking them.

Via Marginal Revolution, a research paper describes the behavior of people diagnosed with borderline personality disorder in a repeated trust game. The gist of it is that cooperation broke down because the subjects made no attempt to restore the counterpart’s trust, even as her willingness to lend deteriorated.

Very interesting stuff, especially in light of the current credit crisis – how much of our economy is dependent on fragile cooperation mechanisms?

Testing Many-Worlds Quantum Theory By Measuring Pattern Convergence Rates

http://arxiv.org/abs/0809.4422

This was published recently, but it seems to have received very few discussion. The paper is really short (2 pages, without abstract and references it would be a single page) and claims to use Bayesian theory to provide a testable formula that should either prove or discard Many Worlds.

I didn’t understand it, but I suspect Tipler may be trying to measure, in one world, how fast the pattern converges summed over many worlds, which I think would be a mistake.

I also suspect Tipler’s ideas won’t be given as careful an examination as they would have if someone else had put them forward.

Is there a physicist in the house?

Phil: what he claims to be an “outline” of a proof really doesn’t say how he gets the result. It’s only that one paragraph, the following paragraphs introduce the terminology for eq (1), they aren’t part of the “outline”.

He does say:
“before measurements, identical copies of the observer exist in parallel universes”
– (which is a not at all the conventional way to think of many worlds, but probably would not lead to an incorrect result in this case, although it would in an epr experiment)
“a Bayesian probability density … is NOT a relative frequency”
– (but by repeating the experiment over and over the relative frequency interpretation would come to the same result; Tipler doesn’t seem to realize that you can repeat the whole experiment, not just have repeated observations in one experiment)

I suspect that Tipler does make the mistake you suggested he might have made, though.

Anyway, he’s wrong in stating that there would be a difference between many worlds and Copenhagen in this case, and his result in eq(1) is clearly wrong for any interpretation.

A question, is Economics about predicting the economy?