Monthly Archives: February 2009

Signaling Math

By Robin Hanson · February 11, 2009 10:00 pm · Comments (39)

It has been two years since I posted a summary of how signaling works; recent discussions suggest maybe I should try again.  Be warned; this time I'll use more math.

Consider authors who must choose a level of emotion e for their writing, given their propaganda factor p, which says how much they care about persuading readers, relative to informing readers.  Assume that authors prefer to be perceived by readers has having low propaganda p, and that everyone knows that m is the maximum possible value of p.  

Assume authors maximize a utility,

U(e;p) = -0.5(e-p)2 – E[p|e] ,

where E[p|e] is the reader estimate of author propaganda p, after having observed author emotion e.  The first term says emotion is more useful for propaganda authors, but the second term says using more emotion may tip off readers to such author intentions.

If readers already knew author propaganda p, signaling would not be an issue, and authors would just choose e = p.   However, if readers do not fully know author propaganda factor p, then if readers always make exactly the rational inference from observing emotion levels e, and if authors always exactly maximize their utility given this reader behavior (and if we use standard game theory refinements), then the equilibrium satisfies

p = e + 1 – exp(e-m) .

The worst possible author with p = m chooses e = p, just as if signaling were not an issue.  But all other authors choose e < p, asymptotically approaching e = p – 1.  The choice of emotion e fully reveals propaganda p, and everyone but the worst possible type p = m uses less emotion than they would if signaling were not an issue.

So when I suggest that the reason engineers, lawyers, accountants, and academics try to avoid emotion is that they want to be believed by skeptical readers, it is not enough to argue that someone concerned only with imparting as much info as possible to trusting readers would use lots more emotion than such authors do.  The whole point is that reader trust cannot be assumed.

(Note the equilibrium above applies for any info readers have about author propoganda p, as long as their posterior given that info has support up to m.)

GD Star Rating
loading...
Tagged as:

Cynicism in Ev-Psych (and Econ?)

By Eliezer Yudkowsky · February 11, 2009 10:06 am · Comments (35)

Though I know more about the former than the latter, I begin to suspect that different styles of cynicism prevail in evolutionary psychology than in microeconomics.

Evolutionary psychologists are absolutely and uniformly cynical about the real reason why humans are universally wired with a chunk of complex purposeful functional circuitry X (e.g. an emotion) – we have X because it increased inclusive genetic fitness in the ancestral environment, full stop.

Evolutionary psychologists are mildly cynical about the environmental circumstances that activate and maintain an emotion.  For example, if you fall in love with the body, mind, and soul of some beautiful mate, an evolutionary psychologist would like to check up on you in ten years to see whether the degree to which you think your mate's mind is still beautiful, correlates with independent judges' ratings of how physically attractive that mate still is.

But it wouldn't be conventionally ev-psych cynicism to suppose that you don't really love your mate, and that you were actually just attracted to their body all along, but that instead you told yourself a self-deceiving story about virtuously loving them for their mind, in order to falsely signal commitment.

Robin, on the other hand, often seems to think that this general type of cynicism is the default explanation and that anything else bears a burden of proof – why suppose an explanation that invokes a genuine virtue, when a selfish desire will do?

Of course my experience with having deep discussions with economists mostly consists of talking to Robin, but I suspect that this is at least partially reflective of a difference between the ev-psych and economic notions of parsimony.

Ev-psychers are trying to be parsimonious with how complex of an adaptation they postulate, and how cleverly complicated they are supposing natural selection to have been.

Economists… well, it's not my field, but maybe they're trying be parsimonious by having just a few simple motives that play out in complex ways via consequentialist calculations?

Continue reading "Cynicism in Ev-Psych (and Econ?)" »

GD Star Rating
loading...

Informers and Persuaders

By Eliezer Yudkowsky · February 10, 2009 3:22 pm · Comments (17)

Suppose we lived in this completely alternate universe where nothing in academia was about status, and no one had any concept of style.  A universe where people wrote journal articles, and editors approved them, without the tiniest shred of concern for what "impression" it gave – without trying to look serious or solemn or sophisticated, and without being afraid of looking silly or even stupid.  We shall even suppose that readers, correspondingly, have no such impressions.

In this simpler world, academics write papers from only two possible motives:

First, they may have some theory of which they desire to persuade others; this theory may or may not be true, and may or may not be believed for virtuous reasons or with very strong confidence, but the writer of the paper desires to gain adherents for it.

Second, there will be those who write with an utterly pure and virtuous love of the truthfinding process; they desire solely to give people more unfiltered evidence and to see evidence correctly added up, without a shred of attachment to their or anyone else's theory.

People in the first group may want to signal membership in the second group, but people in the second group only want their readers to be well-informed.  In any case, to first order we must suppose that none of this is about signaling – that all such motives are just blanked out.

What do journal articles in this world look like, and how do the Persuaders' articles differ from the Informers'?

Continue reading "Informers and Persuaders" »

GD Star Rating
loading...

Against Propaganda

By Robin Hanson · February 9, 2009 7:00 pm · Comments (43)

Communications we read or see produce four kinds of changes in our beliefs:

  • Random – uncorrelated with much else of interest,
  • Info – more correlated with the world as it is, vs. as it might be,
  • Persuasion – more correlated with beliefs authors prefer us to have, and
  • Other – correlations with anything else of interest.

When choosing what to read (or see) how carefully, many of us prefer info to persuasion, and weakly dislike random and other changes. So we watch for signals indicating lots of info relative to persuasion. In contrast, readers who prefer persuasion over info seek signals indicating their favored mixture.

For example, consider contexts where people reaffirm their religious and patriotic allegiances, where coaches inspire teams or warriors inspire troops, or where "inspirational" speakers persuade folks to stick to their diets, try harder to succeed in their careers, or hold out for their romantic ideals. In such propaganda contexts, impressive charismatic leaders tend to speak in simple repetitive eloquent poetic vague emotional language, often with rambling structures, engaging stories, vivid colorful flashy emotional music and visual aids, and artistic impressive comforting communal surroundings.

In contrast, when possibly-hostile and expert critics are addressed by lawyers supporting clients, engineers presenting designs, accountants presenting financial accounts, or academics presenting analyses, styles are more "no-nonsense."  They avoid colorful flashy emotional visual aids and music, use precise concise technical and unemotional language, make structured and standardized arguments, explicitly summarize and address opposing views, make methods and premises explicit, and warn early of conclusions and structures.

These differing styles occur not just because differing communication contexts have differing style requirements, but also because authors try to credibly signal their intentions. Authors who want to be seen as minimizing the propaganda element of their communications avoid using flashy styles, eloquent language, or compelling stories, even when such things would make it easier for readers to assimilate the presented info. After all, readers who cannot easily see that deviating from the usual non-nonsense style here actually promotes info may think worse of them.  One must furthermore worry about being quoted out of context by hostile parties.

Continue reading "Against Propaganda" »

GD Star Rating
loading...
Tagged as: ,

(Moral) Truth in Fiction?

By Eliezer Yudkowsky · February 9, 2009 12:26 pm · Comments (40)

A comment by Anonymous on Three Worlds Collide:

After reading this story I feel myself agreeing with Eliezer more on his views and that seems to be a sign of manipulation and not of rationality.

Philosophy expressed in form of fiction seems to have a very strong effect on people – even if the fiction isn't very good (ref. Ayn Rand).

Robin has similar qualms:

Since people are inconsistent but reluctant to admit that fact, their moral beliefs can be influenced by which moral dilemmas they consider in what order, especially when written by a good writer. I expect Eliezer chose his dilemmas in order to move readers toward his preferred moral beliefs, but why should I expect those are better moral beliefs than those of all the other authors of fictional moral dilemmas?

If I'm going to read a literature that might influence my moral beliefs, I'd rather read professional philosophers and other academics making more explicit arguments.

I replied that I had taken considerable pains to set out the explicit arguments before daring to publish the story.  And moreover, I had gone to considerable length to present the Superhappy argument in the best possible light.  (The opposing viewpoint is the counterpart of the villain; you want it to look as reasonable as possible for purposes of dramatic conflict, the same principle whereby Frodo confronts the Dark Lord Sauron rather than a cockroach.)

Robin didn't find this convincing:

I don't think readers should much let down their guard against communication modes where sneaky persuasion is more feasible simply because the author has made some more explicit arguments elsewhere…  Academic philosophy offers exemplary formats and styles for low-sneak ways to argue about values.

I think that this understates the power and utility of fiction.  I once read a book that was called something like "How to Read" (no, not "How to Read a Book") which said that nonfiction was about communicating knowledge, while fiction was about communicating experience.

Continue reading "(Moral) Truth in Fiction?" »

GD Star Rating
loading...

…And Say No More Of It

By Eliezer Yudkowsky · February 8, 2009 7:15 pm · Comments (19)

Followup to:  The Thing That I Protect

Anything done with an ulterior motive has to be done with a pure heart.  You cannot serve your ulterior motive, without faithfully prosecuting your overt purpose as a thing in its own right, that has its own integrity.  If, for example, you're writing about rationality with the intention of recruiting people to your utilitarian Cause, then you cannot talk too much about your Cause, or you will fail to successfully write about rationality.

This doesn't mean that you never say anything about your Cause, but there's a balance to be struck.  "A fanatic is someone who can't change his mind and won't change the subject."

In previous months, I've pushed this balance too far toward talking about Singularity-related things.  And this was for (first-order) selfish reasons on my part; I was finally GETTING STUFF SAID that had been building up painfully in my brain for FRICKIN' YEARS.  And so I just kept writing, because it was finally coming out.  For those of you who have not the slightest interest, I'm sorry to have polluted your blog with that.

When Less Wrong starts up, it will, by my own request, impose a two-month moratorium on discussion of "Friendly AI" and other Singularity/intelligence explosion-related topics.

There's a number of reasons for this.  One of them is simply to restore the balance.  Another is to make sure that a forum intended to have a more general audience, doesn't narrow itself down and disappear.

But more importantly – there are certain subjects which tend to drive people crazy, even if there's truth behind them.  Quantum mechanics would be the paradigmatic example; you don't have to go funny in the head but a lot of people do.  Likewise Godel's Theorem, consciousness, Artificial Intelligence -

The concept of "Friendly AI" can be poisonous in certain ways.  True or false, it carries risks to mental health.  And not just the obvious liabilities of praising a Happy Thing.  Something stranger and subtler that drains enthusiasm.

Continue reading "…And Say No More Of It" »

GD Star Rating
loading...

The Thing That I Protect

By Eliezer Yudkowsky · February 7, 2009 2:18 pm · Comments (19)

Followup to:  Something to Protect, Value is Fragile

"Something to Protect" discursed on the idea of wielding rationality in the service of something other than "rationality".  Not just that rationalists ought to pick out a Noble Cause as a hobby to keep them busy; but rather, that rationality itself is generated by having something that you care about more than your current ritual of cognition.

So what is it, then, that I protect?

I quite deliberately did not discuss that in "Something to Protect", leaving it only as a hanging implication.  In the unlikely event that we ever run into aliens, I don't expect their version of Bayes's Theorem to be mathematically different from ours, even if they generated it in the course of protecting different and incompatible values.  Among humans, the idiom of having "something to protect" is not bound to any one cause, and therefore, to mention my own cause in that post would have harmed its integrity.  Causes are dangerous things, whatever their true importance; I have written somewhat on this, and will write more about it.

But still – what is it, then, the thing that I protect?

Friendly AI?  No – a thousand times no – a thousand times not anymore.  It's not thinking of the AI that gives me strength to carry on even in the face of inconvenience.

Continue reading "The Thing That I Protect" »

GD Star Rating
loading...

Our Biggest Surprise

By Robin Hanson · February 6, 2009 11:30 am · Comments (75)

We as a civilization know a lot more today than a random hunter-gatherer from fifty thousand years ago knew.  At a fun dinner with Cosmic Variances's thoughtful Sean Carroll last night, I asked:  What have we learned that is the most surprising?  Sean initially answered "quantum mechanics" but I complained that bundles together too many different things we've learned; I instead want to know what single feature of have we learned would most surprise our distant ancestors? 

Sean then suggested non-determinism, that quantum mechanics appears to suggest that the past does not determine the future.  I suggested what would most surprise our distant ancestors is how big is our universe.  It is big in time and space, in extent and detail, and in the range of things that can fill this extended detailed spacetime. 

So what would you say has been our biggest surprise, weighing not just raw info but also that info's relevance? 

Added:  OK, I see two related surprises, one empirical and one logical. The empirical surprise is that the universe really is big.  The logical surprise is that a big enough universe with a small number of simple essenses can reproduce all of the complex local phenomena that one might otherwise explain via design or a large number of essences.  So per Eliezer and Julian, enough inanimate objects can produce animate object behavior, and per Jed enough incremental adjustments can produce bio and social order.

More added:  Sean remembers the conversation a bit differently; he's probably right. He also asks "the complementary question: what is the most surprising thing about the universe that we haven’t yet discovered, but plausibly could?"

GD Star Rating
loading...
Tagged as:

Epilogue: Atonement (8/8)

By Eliezer Yudkowsky · February 6, 2009 6:52 am · Comments (72)

(Part 8 of 8 in "Three Worlds Collide")

Fire came to Huygens.

The star erupted.

Stranded ships, filled with children doomed by a second's last delay, still milled around the former Earth transit point.  Too many doomed ships, far too many doomed ships.  They should have left a minute early, just to be sure; but the temptation to load in that one last child must have been irresistable.  To do the warm and fuzzy thing just this one time, instead of being cold and calculating.  You couldn't blame them, could you…?

Yes, actually, you could.

The Lady Sensory switched off the display.  It was too painful.

On the Huygens market, the price of a certain contract spiked to 100%.  They were all rich in completely worthless assets for the next nine minutes, until the supernova blast front arrived.

"So," the Lord Pilot finally said.  "What kind of asset retains its value in a market with nine minutes to live?"

Continue reading "Epilogue: Atonement (8/8)" »

GD Star Rating
loading...

Share likelihood ratios, not posterior beliefs

By Anna Salamon and Steve Rayhawk · February 5, 2009 6:19 pm · Comments (32)

When I think of Aumann's agreement theorem, my first reflex is to average.  You think A is 80% likely; my initial impression is that it's 60% likely.  After you and I talk, maybe we both should think 70%.  "Average your starting beliefs", or perhaps "do a weighted average, weighted by expertise" is a common heuristic.

But sometimes, not only is the best combination not the average, it's more extreme than either original belief.

Let's say Jane and James are trying to determine whether a particular coin is fair.  They both think there's an 80% chance the coin is fair.  They also know that if the coin is unfair, it is the sort that comes up heads 75% of the time.

Jane flips the coin five times, performs a perfect Bayesian update, and concludes there's a 65% chance the coin is unfair.  James flips the coin five times, performs a perfect Bayesian update, and concludes there's a 39% chance the coin is unfair.  The averaging heuristic would suggest that the correct answer is between 65% and 39%.  But a perfect Bayesian, hearing both Jane's and James's estimates – knowing their priors, and deducing what evidence they must have seen - would infer that the coin was 83% likely to be unfair.  [Math footnoted.]

Perhaps Jane and James are combining this information in the middle of a crowded tavern, with no pen and paper in sight.  Maybe they don't have time or memory enough to tell each other all the coins they observed.  So instead they just tell each other their posterior probabilities – a nice, short summary for a harried rationalist pair.  Perhaps this brevity is why we tend to average posterior beliefs.

However, there is an alternative.  Jane and James can trade likelihood ratios.  Like posterior beliefs, likelihood ratios are a condensed summary; and, unlike posterior beliefs, sharing likelihood ratios actually works.

Continue reading "Share likelihood ratios, not posterior beliefs" »

GD Star Rating
loading...
Tagged as: , ,