Tag Archives: Disagreement

Who Wants Accuracy?

A well-connected reporter (who I promised I’d keep anonymous) just told me that a major Washington media organization started a project studying major media pundits, and a big part of this project was assessing individual pundit forecast track records. After several months of several folks working on the project, it was killed, supposedly because management decided readers don’t care as much about pundit accuracy as they’d previous thought.

Of course that need not have been their real reason – perhaps some folks didn’t like the ratings it was giving to their favorite pundits. Or perhaps it died for any of a hundred random reasons projects are killed. Even so, I found this anecdote interesting.

GD Star Rating
loading...
Tagged as: , ,

Seek Criticism

Two weeks ago I read Penrose’s new book Cycles of Time. I enjoyed his review of the time’s arrow puzzle, and was intrigued by his proposal that distances fade away in vast infinite futures, allowing them to become tiny flat big bangs again. But not only did Penrose wave his arms pretty wildly on how there could be a metric along which metrics would disappear in approaching the vast-tiny border, he seems to make a very elementary mistake in positing that entropy could have a similar magnitude in our big bang post and our vast distant future, because info is lost in evaporating black holes. The entropy in black hole radiation is more than the holes themselves, which is far more than a tiny flat big bang before.

Raphael Bousso (co-author of that Anthropic breakthrough I raved about in ’08) reviews the book in Science, and seems to agree:

Penrose is at his best when he explains this deep and beautiful mystery, and the book may be worth reading for this chapter alone. However, he compounds the shortcomings of his cyclic universe model when he argues that it can solve the low-entropy problem. At this point, another idea is introduced: like vacuum cleaners, black holes appear to reduce disorder by swallowing matter. By the end of one “aeon,” Penrose argues, most matter has ended up in giant black holes. Very little entropy remains, and the next aeon can commence in perfect order. The second law guarantees that a vacuum cleaner does not actually decrease the overall disorder; at best, it just shifts it around. In fact, the machine creates far more entropy than it destroys (for example, by heating up the air in the room). A black hole, it turns out, is not different. Penrose’s assertion that black holes destroy entropy is flatly contradicted by “the generalized second law of thermodynamics”. (more)

How could such a big-shot make such a simple mistake? One should seriously consider the possibility that he isn’t saying what he appears to be saying, and in fact is saying something much more clever and insightful. But if so why wouldn’t he have devoted more effort to explaining, to avoid the misunderstanding. His book reads as if he didn’t even consider that this criticism would be offered. And that fact leads me to believe Penrose considers himself to be such a big shot that he didn’t even ask colleagues to read and criticize his book before publication. And that sort of isolation makes me more willing to believe that he did in fact just make a simple mistake.

The lesson: no matter how much better you think you are than the lowly incompetents that surround you, you’d still do well to ask for and listen to criticism.

GD Star Rating
loading...
Tagged as: ,

Against DIY Academics

Vladimir M:

In many areas, … respectable academic authors are the richest and most reliable source of information, and people claiming things completely outside the academic mainstream are almost certain to be crackpots. … [But] what happens when a field fails both of them, having no clear research directions and at the same time being highly relevant to ideologues and interest groups? Unsurprisingly, it tends to be really bad.

The clearest example of such a field is probably economics, particularly macroeconomics. … Even a casual inspection of the standards in this field shows clear symptoms of cargo-cult science: weaving complex and abstruse theories that can be made to predict everything and nothing, manipulating essentially meaningless numbers as if they were objectively measurable properties of the real world, experts with the most prestigious credentials dismissing each other as crackpots.

Now it might be that academics in fields low in ideological conflict tend to accept each other’s work too easily, to protect the reputation of their field. In this case, fields with more ideological conflict could be more reliable, if that conflict led to more critical examination of results.

However, let us accept for the sake of argument that all else equal in ideological fields intellectual progress is slower, and claims tend to be make with more overconfidence.  What exactly would this imply for your beliefs about this area?

It certainly wouldn’t imply that you ignore what experts write. Yes, it makes sense to adjust your beliefs for the average overconfidence there, but even with large adjustments your best estimates should still rely heavily on average expert estimates.  After all, even if they know less than they think, they still know a lot more than you.

I suspect that what Vladimir and others usually have in mind is Do It Yourself Science:

Looking at the data yourself and drawing your own conclusions.

Now trying your own hand at the subject can help you to understand most any subject.  It can help you discern who are the real experts, and better understand what they say.  There’s a reason students are asked to do labs and problem sets.

But if you plan to mostly ignore the experts and base your beliefs on your own analysis, you need to not only assume that ideological bias has so polluted the experts as to make them nearly worthless, but you also need to assume that you are mostly immune from such problems!

Yes, this is a natural assumption to make, as we rarely feel that we are subject to the biases we suspect we see in others.  But without substantial evidence clearly supporting it, this is mostly just wishful thinking.  If ideology severely compromises others’ analysis on this subject, then most likely it severely comprises yours as well.  You should mostly just avoid having opinions on the subject.  But if you must have reliable opinions, average expert opinions are probably still your best bet.  (Unless of course you have a prediction market available. :) )

GD Star Rating
loading...
Tagged as: ,

Beware “Consensus”?

If your doctor discourages you from seeking another opinion, you have even more reason to get one. (more)

Honest contrarians who expect reasonable outsiders to give their contrarian view more than normal credence should point to strong outside indicators that correlate enough with contrarians tending more to be right. (more)

Perhaps one strong outside indicator that a contrarian view is right is when the media goes out of its way to say that it is opposed by a “scientific consensus”! Ron Bailey in July:

Several [out of the eight media-declared] scientific consensuses before 1985 turned out to be wrong or exaggerated, e.g., saccharin, dietary fiber, fusion reactors, stratospheric ozone depletion, and even arguably acid rain and high-dose animal testing for carcinogenicity.

It seems to me that for folks with a contrarian bent, getting more better studies like this should be a high priority. More details from Ron: Continue reading "Beware “Consensus”?" »

GD Star Rating
loading...
Tagged as: ,

What Is “Belief”?

Richard Chappell has a couple of recent posts on the rationality of disagreement. As this fave topic of mine appears rarely in the blogsphere, let me not miss this opportunity to discuss it.

In response to the essential question “why exactly should I believe I am right and you are wrong,” Richard at least sometimes endorses the answer “I’m just lucky.” This puzzled me; on what basis could you conclude it is you and not the other person who has made a key mistake? But talking privately with Richard, I now understand that he focuses on what he calls “fundamental” disagreement, where all parties are confident they share the same info and have made no analysis mistakes.

In contrast, my focus is on cases where parties assume they would agree if they shared the same info and analysis steps.  These are just very different issues, I think.  Unfortunately, they appear to be more related than they are, because of a key ambiguity in what we mean by “belief.”  Many common versions of this concept do not “carve nature at the relevant joints.”  Let me explain.

Every decision we make is influenced by a mess of tangled influences that can defy easy classification. But one important distinction, I think, is between (A) influences that come most directly from inside of us, i.e., from who we are, and (B) influences that come most directly from outside of us. (Yes, of course, indirectly each influence can come from everywhere.) Among outside influences, we can also usefully distinguish between (B1) influences which we intend to track the particular outside things that we are reasoning about, from (B2) influences that come from rather unrelated sources.

For example, our attitude toward rain soon might be influenced by (A) our dark personality, that makes us expect dark things, and from (B1) seeing dark clouds, which is closely connected to the processes that make rain.  Our attitude toward rain might also be influenced by (B2) broad social pressures to make weather forecasts match the emotional mood of our associates, even when this has little relation to if there will be rain.

Differing attitudes between people on rain soon is mainly problematic regarding (B1) aspects of our mental attitudes which we intend to have track that rain. Yes of course if we are different inside, and are ok with remaining different in such ways, then it is ok for our decisions to be influenced by such differences. But such divergence is not so ok regarding the aspects of our minds that we intend to track things outside our minds.

Imagine that two minds intend for certain aspects of their mental states to track the same outside object, but then they find consistent or predictable differences between their designated mental aspects. In this case these two minds may suspect that their intentions have failed. That is, their disagreement may be evidence suggesting that for at least one of them other influences have contaminated mental aspects that person had intended would just track that outside object.

This is to me the interesting question in rationality of disagreement; how do we best help our minds to track the world outside us in the face of apparent disagreements? This is just a very different question from what sort of internal mental differences we are comfortable with having and acknowledging.

Unfortunately most discussion about “beliefs” and “opinions” are ambiguous regarding whether those who hold such things intend for them to just be mental aspects that track outside objects, or whether such things are intended to also reflect and express key internal differences. Do you want your “belief” in rain to just track the chance it will rain, or do you also want it to reflect your optimism toward life, your social independence, etc.?  Until one makes more clear what mental aspects exactly are referred to by the word “belief”, it seem very hard to answer such questions.

This ambiguity also clouds our standard formal theories. Let me explain.  In standard expected-utility decision theory, the two big influences on actions are probabilities and utilities, with probabilities coming from a min-info “prior” plus context-dependent info. Most econ models of decision making assume that all decision makers use expected utility and have the same prior. For example, agents might start with the same prior, get differing info about rain, take actions based on their differing info and values, and then change their beliefs about rain after seeing the actions of others. In such models, info and thus probability is (B1) what comes from outside agents to influence their decisions, while utility (A) comes from inside. Each probability is designed to be influenced only by the thing it is “about,” minimizing influence from (A) internal mental features or (B2) unrelated outside sources.

In philosophy, however, it is common to talk about the possibility that different people have differing priors. Also, for every set of consistent decisions one could make, there are an infinite number of different pairs of probabilities and utilities that produce those decisions. So one can actually model any situation with several expected-utility folks making decisions as either one with common priors or with uncommon priors.

Thus in contrast to the practice of most economists, philosophers’ use of “belief” (and “probability” and “prior”) confuses or mixes (A) internal and (B) external sources of our mental states. Because of this, it seems pointless for me to argue with philosophers about whether rational priors are common, or whether one can reasonably have differing “beliefs” given the same info and no analysis mistakes. We would do better to negotiate clearer language to talk about the parts of our mental states that we intend to track what our decisions are about.

Since I’m an economist, I’m comfortable with the usual econ habit of using “probability” to denote such outside influences intended to track the objects of our reasoning.  (Such usage basically defines priors to common.) But I’m willing to cede words like “probability”, “belief” or “opinion” to other purposes, if other important connotations need to be considered.

However, somewhere in our lexicon for discussing mental states we need words to refer to something like what econ models usually mean by “probabilities”, i.e., aspects of our mental states that we intend to track the objects of our reasoning, and to be minimally influenced by other aspects of our mental states.

(Of course all this can be applied to “beliefs” about our own minds, if we consider influences coming from our minds as if it were something outside.)

GD Star Rating
loading...
Tagged as: , ,

The Wisdom Of Others

When we choose to act on our own private clues, or to infer the clues of others from their actions, we too often neglect the wisdom of others:

In situations where it is empirically optimal to follow others and contradict one’s own information, the players err in the majority of cases, forgoing substantial parts of earnings. The average player contradicts her own signal only if the empirical odds ratio of the own signal being wrong, conditional on all available information, is larger than 2:1, rather than 1:1 as would be implied by rational expectations. … In total, the meta dataset contains 29,923 decisions made by 2,813 participants in 13 studies. All participants observe a private signal and a (possibly empty) string of previous choices made by others in analogous situations. In all decision problems there are two actions and two possible payoffs, but the dataset nevertheless comprises a large variety in environments, instructions, players’ personal characteristics, and histories of other players’ choices. (more)

Of course copying others’ acts sends a bad signal about our confidence in own own info and and analysis abilities. So it can make sense to focus more on one’s own clues to the extent is is important to send a positive signal to observers.  Just beware of assuming too easily that such gains are substantial.

GD Star Rating
loading...
Tagged as: ,

Tug-O-War Is Not Charity

Arnold Kling:

Ezra Klein thinks that political organizations are worthwhile charities.

If you donate money to a food bank, it can provide only as much food as your money can buy. If you donate it to a nonprofit that specializes in food policy issues, it can persuade legislators to pass a new program – or reform an existing one – that can do much more than any single food bank.

So he winds up giving his money to support a think tank whose employees are somewhere around the 95th percentile of the income distribution, in the hope that they will help tilt the rent-seeking in Washington in a direction that he likes. … It is actually sort of sad for a policy wonk to settle on the idea of making donations to an organization of policy wonks.

If public policy is a point in a high dimensional space, then every policy change has two components: a partisan and a non-partisan change. Partisan changes are along standard partisan axes, where people are lined up in a tug-o-war on different sides pulling in different directions. Non partisan changes, in contrast, are not seen as a win for one side relative to others. Technically, partisan changes project total changes into the partisan subspace.

Assuming all parties think they seek good, partisan changes can only be good if some parties are right while others are wrong about what is good. In contrast, you can be right about a non-partisan change without others being wrong. Since the total space has a far larger dimension that the partisan space, there is a huge scope for searching in that larger space for changes that all sides could see as good. And donations to encourage such efforts can indeed consistently produce large social gains relative to their costs.

Donations to change policy within the partisan subspace, however, only achieve good when they happen to be on the right side of partisan disagreements. Averaged over the disagreeing parties, such donations cannot on average achieve good unless there is a correlation between between donations, or donation effectiveness, and which sides are right.  Even if you think you are right at the moment on your particular partisan policy opinions, you can’t think it good on average to encourage partisan donations, unless you think donations tend overall to go to the good or more donation-effective sides.

Unfortunately most thinktank efforts go into pushing for their sides within the partisan subspace, because that is what most donors care about. For example, Ezra’s two concrete policy examples, of “the need for food banks and homeless shelters and social services” and “repeal the 2010 health-care reform legislation,” are both clearly partisan.

Humans clearly tend to be overconfident about politics. Since you are human, that tendency is a likely cause of your confidence in your political opinions. If your politics were about doing good with policy, you should correct for that overconfidence, and that correction would on average move folks to have little interest in partisan pushes.  Of course if your politics is not about policy, but about showing loyalty, how clever or informed you are, etc., well then go right ahead and be partisan. But don’t tell me that is generally beneficial charity.

GD Star Rating
loading...
Tagged as: , ,

Opinion Warning Signs

Signs that your opinions function more to signal loyalty and ability than to estimate truth:

  1. You find it hard to be enthusiastic for something until you know that others oppose it.
  2. You have little interest in getting clear on what exactly is the position being argued.
  3. Realizing that a topic is important and neglected doesn’t make you much interested.
  4. You have little interest in digging to bigger topics behind commonly argued topics.
  5. You are less interested in a topic when you don’t foresee being able to talk about it.
  6. You are uncomfortable taking a position near the middle of the opinion distribution.
  7. You are uncomfortable taking a position of high uncertainty about who is right.
  8. You care far more about current nearby events than similar distant or past/future events.
  9. You find it easy to conclude that those who disagree with you are insincere or stupid.
  10. You are reluctant to change your publicly stated positions in response to new info.
  11. You are reluctant to agree a rival’s claim, even if you had no prior opinion on the topic.
  12. You are reluctant to take a position that raises the status of rivals.
  13. You care more about consistency between your beliefs than about belief accuracy.
  14. You go easy on sloppy arguments by folks on “your side.”
  15. You have little interest in practical concrete implications of commonly argued topics.
  16. Your opinion doesn’t much change after talking with smart folks who know more.
  17. You are especially eager to drop names when explaining positions and arguments.
  18. You find it hard to list weak points and counter-arguments on your positions.
  19. You feel passionately about a topic, but haven’t sought out much evidence.
  20. You are reluctant to not have an opinion on commonly discussed topics.
  21. More?

Of course you may want your opinions to mainly signal loyalty and ability.

GD Star Rating
loading...
Tagged as: ,

Friends Disagree Lots

Seems we assume our friends agree with us, just because they are our friends.  Yes you agree with your friends more than with random folk, but you agree less than you think:

Friends disagree more than they think they do. In particular, friends are typically unaware of their disagreements, even when they say they discuss the topic, suggesting that discussion is not the primary means by which friends infer each other’s views on particular issues. Rather, it appears that respondents infer opinions in part by relying on stereotypes of their friends and in part by projecting their own views. (more)

GD Star Rating
loading...
Tagged as:

Disagreement Is Far

Key sources of disagreement among economic forecasters are identified by using data on … forecasters’ long- and short-run predictions of macroeconomic variables. Dispersion among forecasters is highest at long horizons where private information is of limited value and lower at short forecast horizons. Moreover, differences in views persist through time. Such differences in opinion cannot be explained by differences in information sets; our results indicate they stem from heterogeneity in priors or models. Differences in opinion move countercyclically, with heterogeneity being strongest during recessions where forecasters appear to place greater weight on their prior beliefs. (more)

These authors speak sloppily.  Their results suggest that disagreements on the state of the economy cannot be attributed much to differing “near” late-breaking info of the sort one usually feeds into models that predict the state of the economy.  But they could be due to differing “far” big-picture info of the sort that leads one to prefer some such models over others.  Disagreement is indeed far.

GD Star Rating
loading...
Tagged as: , ,