Overcoming Bias

Singletons Rule OK

Reply to:  Total Tech Wars

How does one end up with a persistent disagreement between two rationalist-wannabes who are both aware of Aumann's Agreement Theorem and its implications?

Such a case is likely to turn around two axes: object-level incredulity ("no matter what AAT says, proposition X can't really be true") and meta-level distrust ("they're trying to be rational despite their emotional commitment, but are they really capable of that?").

So far, Robin and I have focused on the object level in trying to hash out our disagreement.  Technically, I can't speak for Robin; but at least in my own case, I've acted thus because I anticipate that a meta-level argument about trustworthiness wouldn't lead anywhere interesting.  Behind the scenes, I'm doing what I can to make sure my brain is actually capable of updating, and presumably Robin is doing the same.

(The linchpin of my own current effort in this area is to tell myself that I ought to be learning something while having this conversation, and that I shouldn't miss any scrap of original thought in it - the Incremental Update technique. Because I can genuinely believe that a conversation like this should produce new thoughts, I can turn that feeling into genuine attentiveness.)

Yesterday, Robin inveighed hard against what he called "total tech wars", and what I call "winner-take-all" scenarios:

Robin:  "If you believe the other side is totally committed to total victory, that surrender is unacceptable, and that all interactions are zero-sum, you may conclude your side must never cooperate with them, nor tolerate much internal dissent or luxury."

Robin and I both have emotional commitments and we both acknowledge the danger of that.  There's nothing irrational about feeling, per se; only failure to update is blameworthy.  But Robin seems to be very strongly against winner-take-all technological scenarios, and I don't understand why.

Among other things, I would like to ask if Robin has a Line of Retreat set up here - if, regardless of how he estimates the probabilities, he can visualize what he would do if a winner-take-all scenario were true.

Beliefs Require Reasons, or: Is the Pope Catholic? Should he be?

In the early days of this blog, I would pick fierce arguments with Robin about the no-disagreement hypothesis.  Lately, however, reflection on things like public reason have brought me toward agreement with Robin, or at least moderated my disagreement.  To see why, it's perhaps useful to take a look at the newspapers...

the pope said the book “explained with great clarity” that “an interreligious dialogue in the strict sense of the word is not possible.” In theological terms, added the pope, “a true dialogue is not possible without putting one’s faith in parentheses.”

What are we to make of a statement like this?

Whence Your Abstractions?

Reply to:  Abstraction, Not Analogy

Robin asks:

Eliezer, have I completely failed to communicate here?  You have previously said nothing is similar enough to this new event for analogy to be useful, so all we have is "causal modeling" (though you haven't explained what you mean by this in this context).  This post is a reply saying, no, there are more ways using abstractions; analogy and causal modeling are two particular ways to reason via abstractions, but there are many other ways.

Well... it shouldn't be surprising if you've communicated less than you thought.  Two people, both of whom know that disagreement is not allowed, have a persistent disagreement.  It doesn't excuse anything, but - wouldn't it be more surprising if their disagreement rested on intuitions that were easy to convey in words, and points readily dragged into the light?

I didn't think from the beginning that I was succeeding in communicating.  Analogizing Doug Engelbart's mouse to a self-improving AI is for me such a flabbergasting notion - indicating such completely different ways of thinking about the problem - that I am trying to step back and find the differing sources of our differing intuitions.

(Is that such an odd thing to do, if we're really following down the path of not agreeing to disagree?)

"Abstraction", for me, is a word that means a partitioning of possibility - a boundary around possible things, events, patterns.  They are in no sense neutral; they act as signposts saying "lump these things together for predictive purposes".  To use the word "singularity" as ranging over human brains, farming, industry, and self-improving AI, is very nearly to finish your thesis right there.

Setting The Stage

As Eliezer and I begin to explore our differing views on singularity, perhaps I should summarize my current state of mind.   

We seem to agree that:

1. Machine intelligence would be a development of almost unprecedented impact and risk, well worth considering now.
2. Feasible approaches include direct hand-coding, based on a few big and lots of little insights, and emulations of real human brains. 
3. Machine intelligence will more likely than not appear with a century, even if the progress rate to date does not strongly suggest the next few decades. 
4. Many people say silly things here, and we do better to ignore them than to try to believe the opposite. 
5. Math and deep insights (especially probability) can be powerful relative to trend-fitting and crude analogies. 
6. Long term historical trends are suggestive of future events, but not strongly so.
7. Some should be thinking about how to create "friendly" machine intelligences. 

We seem to disagree modestly about the relative chances of the emulation and direct-coding approaches; I think the first and he thinks the second is more likely to succeed first.  Our largest disagreement seems to be on the chances that a single hand-coded version will suddenly and without warning change from nearly powerless to overwhelmingly powerful; I'd put it as less than 1% and he seems to put it as over 10%. 

Conformity Shows Loyalty

"The world has too many people showing too much loyalty to their groups.  That is why I'm so proud to be member of ALU, anti-loyalists united, where we refuse to show loyalty to any other groups. My local chapter just kicked out George for suspicion of showing loyalty to California, and we chastised Ellen for expressing doubts about the latest anti-loyalty directives from headquarters.  We'll only lick loyalty by showing we are united behind our courageous ALU leaders.  All hail ALU!"

Sounds pretty silly, right?  But I hear something pretty similar when I hear folks say they are proud to be part of a group that fights conformity by pushing their unusual beliefs.  Especially when such folks seem more comfortable claiming their beliefs contribute to diversity than that they are true.   

We use belief conformity to show loyalty to particular groups, relative to other groups.  We rarely bother to show loyalty to humanity as a whole, because non-humans threaten little.  So we rarely bother to try to conform our beliefs with humanity as a whole, which is why herding experiments with random subjects show no general conformity tendencies. 

Our conformity efforts instead target smaller in-groups, with more threatening out-groups.  And we are most willing to conform our beliefs on abstract ideological topics, like politics or religion, where our opinions have few other personal consequences.  Our choices show to which conflicting groups we feel the most allied.   

You just can't fight "conformity" by indulging the evil pleasure of enjoying your conformity to a small tight-knit group of "non-conformists."  All this does is promote some groups at the expense of other groups, and poisons your mind in the process.  It is like fighting "loyalty" by dogged devotion to an anti-loyalty alliance.

Best to clear your mind and emotions of group loyalties and resentments and ask, if this belief gave me no pleasure of rebelling against some folks or identifying with others, if it was just me alone choosing, would my best evidence suggest that this belief is true?  All else is the road to rationality ruin. 

Disagreement Debate Status?

This blog has had many posts on disagreement, especially early on.  For example, I've posted on the basic idea that we can't foresee to disagree, that we should have common priors and not accept genetic influences, and that this all should apply to logical truths and values.  I discussed specific math models, majoritarianism and meta-majoritarianism, how to share info without disagreeing, and two examples of when to agree and one of when to disagree.   I also give frequent talks on the subject. 

So what do folks think is the status of the debate on the rationality of disagreement?  That is, how reluctant do you think people should typically be to knowingly disagree with one another, and if the arguments I've outlined  seem to have some potential to influence this reluctance, what more is needed to see if they can fulfill this potential?

Crisis of Faith

Followup to:  Make an Extraordinary Effort, The Meditation on Curiosity, Avoiding Your Belief's Real Weak Points

"It ain't a true crisis of faith unless things could just as easily go either way."
        - Thor Shenkel

Many in this world retain beliefs whose flaws a ten-year-old could point out, if that ten-year-old were hearing the beliefs for the first time.  These are not subtle errors we are talking about.  They would be child's play for an unattached mind to relinquish, if the skepticism of a ten-year-old were applied without evasion. As Premise Checker put it, "Had the idea of god not come along until the scientific age, only an exceptionally weird person would invent such an idea and pretend that it explained anything."

And yet skillful scientific specialists, even the major innovators of a field, even in this very day and age, do not apply that skepticism successfully.  Nobel laureate Robert Aumann, of Aumann's Agreement Theorem, is an Orthodox Jew:  I feel reasonably confident in venturing that Aumann must, at one point or another, have questioned his faith.  And yet he did not doubt successfully.  We change our minds less often than we think.

This should scare you down to the marrow of your bones.  It means you can be a world-class scientist and conversant with Bayesian mathematics and still fail to reject a belief whose absurdity a fresh-eyed ten-year-old could see.  It shows the invincible defensive position which a belief can create for itself, if it has long festered in your mind.

What does it take to defeat an error which has built itself a fortress?

But by the time you know it is an error, it is already defeated.  The dilemma is not "How can I reject long-held false belief X?" but "How do I know if long-held belief X is false?"  Self-honesty is at its most fragile when we're not sure which path is the righteous one.  And so the question becomes:

How can we create in ourselves a true crisis of faith, that could just as easily go either way?

Political Parties are not about Policy

Shankar Vedantam explains in the Post why politics isn't about policy:

In 2004 ... die-hards in both parties felt that the choice between George W. Bush and John F. Kerry was much sharper on a host of issues than in any presidential contest going back to 1984.  But when political scientist Marc J. Hetherington quizzed moderates, he found to his surprise that he got the opposite answer. ... If anything, moderates in 2004 saw the Republican and Democratic nominees as being more alike than in any election since 1988.  The schism between moderates and partisans has intensified in this election ...

Hetherington believes that much of the loyalists' perception of a yawning divide has little to do with issues. Rather, he said, what has happened in recent years is that partisans have come to identify with their parties in much the manner that sports fans identify with their teams. The strong views they feel on many issues do not drive their party affiliation; it is their party affiliation that drives their strong views.

Say It Loud

Reply to:  Overconfidence is Stylish

I respectfully defend my lord Will Strunk:

"If you don't know how to pronounce a word, say it loud! If you don't know how to pronounce a word, say it loud!"  This comical piece of advice struck me as sound at the time, and I still respect it. Why compound ignorance with inaudibility?  Why run and hide?

How does being vague, tame, colorless, irresolute, help someone to understand your current state of uncertainty?  Any more than mumbling helps them understand a word you aren't sure how to pronounce?

Goofus says:  "The sky, if such a thing exists at all, might or might not have a property of color, but, if it does have color, then I feel inclined to state that it might be green."

Gallant says:   "70% probability the sky is green."

Which of them sounds more confident, more definite?

But which of them has managed to quickly communicate their state of uncertainty?

(And which of them is more likely to actually, in real life, spend any time planning and preparing for the eventuality that the sky is blue?)

Noble Abstention

High voter turnout need not be good - not only is voting costly, but ignorant voters can make better candidates less likely to win.  Many have noted this last possibility, but I couldn't find any formal models (i.e., of abstention by varying-info-quality voters) - so I made my own.  My results depend on how unequally distributed is voter info; highly unequal info means only a handful should vote, while relatively equal info means most everyone should vote (ignoring voting costs):

Let N voters simultaneously choose to abstain or vote for one of two apriori-equal candidates, based only on which action is more likely to elect the "best" candidate.  Each voter gets an independent private binary signal with a chance (1+q)/2 of accurately marking the best candidate.  Assume that if we rank voters by signal quality q (so the best voter has rank = 1, the next has rank = 2, etc.), we'll find quality and rank are related by a power law, q = q1*rank^-power.  (Voter signal qualities q>0 are common knowledge.)

The following table shows how the number who do not abstain varies with info-quality power, for N = 10,000 voters, q1=0.1, and for two important cases.  In case 1, everyone can choose freely to vote or abstain.  In case 2, each voter chooses if to abstain, assuming no other voter abstains. 

Power	2	1.9	1.8	1.7	1.6	1.5	1.4	1.3	1.2	1.1	1	0.95	0.9	0.85	0.8	0.75	0.7	0.65	0.6	0.55	0.5	0.45
Case 1	1	1	1	1	1	1	1	1	2	2	2	3	3	4	5	6	8	13	24	78	10000	10000
Case 2	109	135	171	221	290	389	532	739	1037	1460	2042	2404	2820	3297	3842	4472	5205	6073	7124	8437	10000	10000

The table shows that for powers above one less than 20% should vote, even if everyone else votes.  And if everyone does what they should, for powers above 1/2, almost no one votes, while for powers below 1/2 everyone votes.  (This last result holds for any N and q1.) 

Similar results probably hold for correlated and non-binary signals.  I'm not sure how we could measure voter info inequality, but I'd give at least two to one odds the effective power is above 1/2, meaning few should vote.  Of course people vote not just to elect the best candidates overall, but to elect folks good for them personally, even when that hurts others.  But this isn't something we should celebrate or encourage, and most voters see themselves as instead voting to improve society overall. 

Large voter turnouts seem to me better understood as overconfidence leading to disagreement - we each think we just know better than others what is good for society.  But too few of us can be right on this for most of us to be (epistemically) rational here.  So I celebrate the noble abstainers, those willing to admit by staying home that their vote would probably just make things worse - we could use a lot more such folks. 

Added 16Sep: Any model with independent signals must either have large electorates get very certain to make the right choice, or must have voter signals get very weak with high rank.  A correlated signals model would be more realistic here.