Who Wants Standards?

Most of us live in worlds of conversation, like books or blogs or chats, where we tend to give many others the benefit of the doubt that they are mostly talking “in good faith.” We don’t just talk to show off or to support allies and knock rivals – we hold our selves to higher standards. But let me explain why that may often be wishful thinking.

I’ve previously suggested that coalition politics infuses a lot of human behavior. That is, we tend to use all available means to try to help “us “and hurt “them”, even if on average these games hurt us all. Coalition politics is a dirt that regularly accumulates in most any corner that is not vigorously and regularly cleaned.

This view predicts that coalition politics also infuses a lot of how writings (and speeches, etc.) are evaluated. That is, when we evaluate the writings of others, we attend to how such evaluations may help our coalitions and hurt rival coalitions. Especially for writings on subjects that have little direct relevance for how we live our lives. Like most topics in most blogs, magazines, journals, books, speeches, etc.

However, while we may find such cynicism plausible as a theory of rivals, we are reluctant to consciously embrace it as theory of ourselves. We instead want to say that we mostly evaluate the writings of others using different criteria. And when we are part of a group that evaluates writings similarly, we want to say this is because our group shares key evaluation criteria beyond “us good, them bad.”

Now some groups can offer concrete evidence for their claims to be relatively clean of coalition politics. These are groups who declare specific “objective” standards to judge writing. That is, they use standards that are relatively easy for outsiders to check. For example, outsiders can relatively easily check groups who evaluate writings based on word count, or on correctness of spelling and grammar. Yes, a commitment to such standards may favor some groups over others, such as good spellers over bad spellers. But it can’t be adjusted very easily to shifting coalitions. Which makes it a poor tool for supporting coalition politics.

Some groups say they judge writings based on their popularity in some audience. And yes, it can be pretty easy to evaluate the popularity of writings. However, it could easily be the audience that is using coalition politics to decide what is popular. Thus using popularity to evaluate writings doesn’t at all ensure that coalition politics doesn’t dominate evaluations.

Some groups claim to evaluate written “maps” based on how well they match intended “territories”. And when it is easy for many clearly-neutral outsiders to visit a territory, it can be easy for outsiders to check that territory-matching is actually how this group evaluates maps. But the harder it is for outsiders to see territories, or to read their supposedly matching maps, and the more easily that outside critics can be credibly accused of political bias, the more easily a group could pretend to evaluate maps based on territory matches, but actually evaluate them via coalition politics. For example, anthropologists watching the private lives of the very rich might write descriptions of those lives that pander to academic presumptions about the very rich, since few academics ever see those lives directly, and the few who do can be accused of biased by association.

Some groups use objective criteria for evaluations, but don’t give those criteria enough weight to stop coalition politics from dominating evaluations. For example, economic theory journals can claim that they only publish articles containing proofs without obvious errors. And the ability of readers to seek errors may ensure that this criteria is usually satisfied. But such journals may still reject most submissions that meet this criteria, allowing coalition politics to dominate which articles are accepted. Winning coalitions may be constrained to include only members capable of constructing proofs without obvious errors, but this need not be very constraining to them.

Another approach is to only use objective evaluation criteria, but to use many such criteria and to be unclear about their relative weights. The more such criteria, the greater the chance of finding criteria to reach whatever evaluation one wants. For example, in many legal areas there is wide agreement on the relevant factors, and on which directions each factor points to in a final decision. Nevertheless, given enough relevant factors, courts may usually have enough discretion to favor either side.

For any one group and their declared criteria of evaluation, it can be hard for outsiders to judge just how much leeway that group has left for coalition politics to influence evaluations. We tend to give the benefit of the doubt to our own groups, but not to rivalrous groups. For example, pro-science anti-religion folks may presume that peer review in scientific journals is mainly used to enforce good evidence norms, but that religious leaders mainly use their discretion in interpreting scriptures to favor their allies.

If they were honest, each group would either declare objective evaluation criteria that leave little room for coalition politics, or accept that outsiders can reasonably presume that coalition politics probably dominates their evaluations. And everyone should expect that even if their group now seems an exception where other criteria dominate, it will probably not remain so for long. Because these are in fact reasonable assumptions in a world where collation politics is a dirt that regularly and rapidly accumulates in any corner not vigorously and regularly cleaned.

Hey there reader, I’m really am talking about you and the worlds of writing where you live. Do you presume that your worlds are mostly dominated by politics, where different coalitions vie to support allies and knock rivals? Or do you see the groups you hang with as holding themselves to higher standards? If higher standards, are they standards that outsiders can easily check on? Or do you in practice mostly have to trust a small group of insiders to judge if standards are met? And if you have to trust insiders, how sure can you be their choices aren’t mostly driven by coalition politics?

Years ago I struggled with this issue, and wondered what evaluation criteria a group could adopt to robustly induce their writings to roughly tract truth on a wide range of topics, and resist the corrupting pressures of coalition politics to say what key audiences want or expect to hear. I was delighted to find that for a wide range of topics open prediction markets offer such robust criteria. Each trade can be an “edit” of the highly-evaluated “writing” that is the current market odds on each topic. Such edits are rewarded or punished via cash for moving the consensus toward or away from the truth.

I had hoped that many groups would be anxious to avoid the appearance that coalition politics may dirty their evaluations, and thus be eager to adopt new standards that can avoid such an appearance. So I hoped that many groups would want to adopt prediction markets, once they were clearly shown to be feasible and practical. Alas, that seems to not be so.

Today’s winning coalitions seem to prefer to let coalition politics continue to determine who wins in each group. This seems like how police departments would like to appear free from corruption, but not enough to actually make their internal affairs departments report to someone other than the chief of police. We are fond of tarring rival groups with the accusation that coalition politics dominates their evaluations, and we are fond of pretending that we are different. But not enough to visibly block that politics.

GD Star Rating
a WordPress rating system
Tagged as: , ,
Trackback URL:
  • Pingback: Recomendaciones | intelib

  • IMASBA

    With coalition policking happening unconsciously how do you prevent it from influencing prediction markets as well? If people truly believe their coalition is right then they will also wager money on that belief.

    • Christopher Ziemian

      And on average, they will be beaten by those who bet honestly. They will thus lose the money that they need to influence market prices, and those who bet honestly will gain money that they need to influence market prices.

      • http://www.gwern.net/ gwern

        > Some groups claim to evaluate written “maps” based on how well they match intended “territories”. And when it is easy for many clearly-neutral outsiders to visit a territory, it can be easy for outsiders to check that territory-matching is actually how this group evaluates maps. But the harder it is for outsiders to see territories, or to read their supposedly matching maps, and the more easily that outside critics can be credibly accused of political bias, the more easily a group could pretend to evaluate maps based on territory matches, but actually evaluate them via coalition politics. For example, anthropologists watching the private lives of the very rich might write descriptions of those lives that pander to academic presumptions about the very rich, since few academics ever see those lives directly, and the few who do can be accused of biased by association.

        Some
        groups claim to evaluate written “maps” based on how well they match
        intended “territories”. And when it is easy for many clearly-neutral
        outsiders to visit a territory, it can be easy for outsiders to check
        that territory-matching is actually how this group evaluates maps. But
        the harder it is for outsiders to see territories, or to read their
        supposedly matching maps, and the more easily that outside critics can
        be credibly accused of political bias, the more easily a group could
        pretend to evaluate maps based on territory matches, but actually
        evaluate them via coalition politics. – See more at:
        http://www.overcomingbias.com/2014/01/who-wants-clean-standards.html#sthash.cRDAVQrj.dpuf
        Some
        groups claim to evaluate written “maps” based on how well they match
        intended “territories”. And when it is easy for many clearly-neutral
        outsiders to visit a territory, it can be easy for outsiders to check
        that territory-matching is actually how this group evaluates maps. But
        the harder it is for outsiders to see territories, or to read their
        supposedly matching maps, and the more easily that outside critics can
        be credibly accused of political bias, the more easily a group could
        pretend to evaluate maps based on territory matches, but actually
        evaluate them via coalition politics. – See more at:
        http://www.overcomingbias.com/2014/01/who-wants-clean-standards.html#sthash.cRDAVQrj.dpuf
        Some
        groups claim to evaluate written “maps” based on how well they match
        intended “territories”. And when it is easy for many clearly-neutral
        outsiders to visit a territory, it can be easy for outsiders to check
        that territory-matching is actually how this group evaluates maps. But
        the harder it is for outsiders to see territories, or to read their
        supposedly matching maps, and the more easily that outside critics can
        be credibly accused of political bias, the more easily a group could
        pretend to evaluate maps based on territory matches, but actually
        evaluate them via coalition politics. – See more at:
        http://www.overcomingbias.com/2014/01/who-wants-clean-standards.html#sthash.cRDAVQrj.dpu

      • http://overcomingbias.com RobinHanson

        Perhaps you are suggesting that the markets would be corrupted via corrupted settlement of who won the bets?

      • http://www.gwern.net/ gwern

        ‘corrupted’ is too narrow, financial, and strong-sounding a term. My point is that prediction markets are themselves a territory/map group and so have concerns about not doing a good job of making the map correspond to the territory due to influence, be it crass bribery or filtered information or weighted judgment.

      • http://overcomingbias.com RobinHanson

        I agree that prediction markets are only useful if outsiders can much more easily check that bet settlement maps match the territories of the bet topics, relative to how easily they could check that betting odds match the best thoughtful estimates about what those odds should be. But I claim that this is often the case.

      • http://www.gwern.net/ gwern

        I don’t think that they generalize to very many questions. Look at the IARPA competition: already they’ve had to revoke, clarify, change, edit, or completely blindside and infuriate people over how many contracts now? And these are extremely narrow objective concrete questions, typically. But debating the scope of prediction markets is perhaps getting a bit offtopic.

      • http://overcomingbias.com RobinHanson

        It costs effort to figure out how to apply any standards to a wide topic area. But for similarly objective standards, prediction markets do well on cost and breadth.

      • http://www.gwern.net/ gwern

        > It costs effort to figure out how to apply standards to a wide topic area.

        Weren’t you complaining that for wide topic areas there are no standards? Doesn’t that indicate that any standards are too expensive or ineffective to be worthwhile?

      • http://overcomingbias.com RobinHanson

        The required costs are only a small fraction of the amount already being spent on these topic areas.

      • http://www.gwern.net/ gwern

        If the costs are so trivial, why haven’t they been paid before?

      • http://overcomingbias.com RobinHanson

        As I suggested in the post, that would hinder using writing evaluations to play coalition politics.

      • Robert Sams

        Anyone with pretensions of objectivity should be willing to make a market on the empirical implications of his explanations. Otherwise, we have no reason to trust that the motivations behind the explanation are search for truth. Robin’s point should be no more controversial than the insistence that reported empirical results are replicable.

        Agreeing on the criteria to settle a bet isn’t a problem to solve for prediction markets, it’s the problem of defining the empirical implications of any theory/model/explanation. If you can’t do it for prediction markets, you can’t do it for empirical tests in general, reason enough to dismiss anyone not willing to undertake the exercise. Specifying the conditions of bet settlement in a way that can be handled by an independent third party should be considered part of the scientific method.

        For the life of me, I don’t understand why this idea is met with such hostility among academics. Coalition politics seems like as good an explanation as any. So why don’t we define some implications of it, set up the test, and invite the doubters to take the other side of the trade ;)

      • Ronfar

        I was just about to make that exact argument – that people would not accept the objectivity of the process that determined who won the bets. How would anyone settle a bet made between Christians and Muslims about the divinity of Jesus? Or, for a more prosaic example, how would you settle a bet on the outcome of the 2000 U.S. Presidential election if you had to make a decision shortly after Election Day? Even today, there’s a way to make a case that Gore really did win the election. (After the Bush v. Gore lawsuit ended, journalists recounted Florida ballots to see if the results would have changed. The lawsuit involved four Florida counties; even if Gore had won it, recounting only those four counties would not have enough of an effect to swing Florida’s vote total in his favor. On the other hand, a statewide recount, which Gore never asked for, actually would have resulted in a Gore victory.)

      • IMASBA

        “And on average, they will be beaten by those who bet honestly.”

        Even if that’s true they’ll still cause a shift. But why do you assume the majority bets honestly? According to Hanson’s theory most people are part of some coalition(s). (Shifted) honesty would only prevail if a minority (compensated for their size) of coalitions have an opinion on the subject (and that’s assuming coalition biases are dominant vs. all other biases combined), but how often does that happen?

      • Christopher Ziemian

        I do not assume that a majority bets honestly. Even if there is only one honest better and a million ideologues trying to game the market, the honest better will quickly part the fools from their gold and gain more money with which to influence market prices than the lot of them put together. (Unless of course, regulators demand that no one person/company can bet more than some set amount in the market, thus severely limiting the power that elite betters wield over market prices.)

      • IMASBA

        The “winner” only gets his money after the market is finished and will then go on to invest that in some other prediction market where he’s probably biased himself this time. What’s the use of this? Aren’t prediction markets supposed to have some use before the they finish? Without that you might as well have someone write a report instead of having a prediction market.

  • http://entitledtoanopinion.wordpress.com TGGP

    I am reminded of how unusually explicit Steve Sailer was recently on how he evaluated Robert Bork’s “The Tempting of America” on the basis of which coalition would be helped/hurt by his advocated manner of judging, and his note that his unprincipled flip-flopping is only clear to him because in hindsight the wrong coalition benefited.

  • Lord

    Isn’t that why most conversation is within coalitions where this is less likely to apply? Not sure there is much point taking politics out of politics as objectivity is not the goal.

    • http://overcomingbias.com RobinHanson

      Coalition politics usually has a fractal structure; each coalition fragments into internal subgroups when possible.

      • Lord

        This would defeat coalitions if as intense as the coalition themselves so these lesser coalitions must soon fade into interests of common discussion where consideration and persuasion must dominate or risk coalition destruction. You may argue they are not objective but I would say they are uncertain or their position would become the coalition position. The smaller the more intense the belief but the larger the more skeptical the opposition or at least the group that doesn’t care.

  • anonymous

    The problem with objective measures of performance is that such measures tend to corrupt the behavior of the people being evaluated. When students are being tested, they start to study for exams, not to actually learn something; when performance of scientists is being measured by their publication history, they start to research things that lead to easy publications.

    If it was possible to find a measure that has perfect correlation with the behavior we want to encourage, there would be no problem. In practice, such measures are hard, if not impossible to find.

    Also I don’t get how prediction markets would do any better than coalition politics (or objective measures), at least in certain topics. Imagine for example setting up a prediction market to evaluate academic publications. The people trading in the market (assuming the idea becomes popular) would be essentially the same people that evaluate the publications now.

  • Ely Spears

    Imagine a spectrum of group membership. On one far end, a person is a hermit or recluse, joining no groups (not even by implicit participation). On the other end of the spectrum, a person joins all groups and participates as earnestly and fully as time and energy constraints will allow.

    If many people are at one of these extremes or the other, it makes coalition politics harder. So one way people could combat coalition politics is to either join no groups or else join all groups.

    I’m not saying this would be easy or a good idea. It just strikes me as an interesting option.

    It is in some ways related to an old post about using exile as a possible legal punishment. It could also be related to immigration choices, sale of citizenship, etc.

    At a meta level, folks probably actively want coalition politics. They aren’t just engaging in it de facto by happening to have been embedded in a culture where it is pervasive. Few folks try very hard to earnestly be a part of a huge range of potentially ideologically opposed groups (for the purpose of not falling prey to coalition politics of their favored group) and few folks dismiss all groups.

    So we both want there to be rivalries and we want to cheer for our side of the rivalry.

    • Ely Spears

      It’s worth it to add that this works in the reverse direction too: if you really want to reduce coalition politics, you should very happily allow anyone, especially the ideologically opposed, to join any of your groups. And you should not cast much judgement on folks who opt to join no groups.

      • Ely Spears

        Sorry for the spam, but this also reminds me of the concept of “complextropy” that Scott Aaronson blogged about a few years ago: .

        This discussion is a kind of complextropy problem of group politics. If everything is entirely mixed (everyone in every group) or nothing is mixed at all (extremely sharp boundaries between everyone) then you’re at one of the two complextropy extremes.

  • efalken

    I have the ability to reason deductively and inductively, but also prejudices that focus that logic more efficiently. I like myself more than others, and I need coalitions to advance my goals. I have an innate sense of ‘the good’ that embodies the Golden Rule and maximizing the welfare of society over the long run.

    Applying these sometimes conflicting objectives is nontrivial, why life is hard. Objectivity is simply one of many desirable things that is merely good in moderation.

  • Pingback: Overcoming Bias : Why Broken Evals?

  • rrb

    How did you learn to deduce the consequences of the hypothesis that something is dominated by coalition politics?

    More importantly, how can I? Is it a mathematical theory, like game theory? Or, is there a book you can read on how coalition politics generally works? Or guidelines somewhere on how to spot coalitions and associated behaviors?

    EDIT: just found your post “The Coalition Politics Hypothesis”, I guess I can start there

  • Ialdabaoth

    I have attempted to express this exact idea myself, many times. Unfortunately, I’ve never been good enough at coalition politics to be taken seriously.

  • Pingback: Maslow suggests that you should aim higher than pump and dump « Random Xpat Rantings

  • Pingback: Markierungen 07/17/2014 | Snippets