Show Outside Critics

Worried that you might be wrong? That you might be wrong because you are biased? You might think that your best response is to study different kinds of biases, so that you can try to correct your own biases. And yes, that can help sometimes. But overall, I don’t think it helps much. The vast depths of your mind are quite capable of tricking you into thinking you are overcoming biases, when you are doing no such thing.

A more robust solution is to seek motivated and capable critics. Real humans who have incentives to find and explain flaws in your analysis. They can more reliably find your biases, and force you to hear about them. This is of course an ancient idea. The Vatican has long had “devil’s advocates”, and many other organizations regularly assign critics to evaluate presented arguments. For example, academic conferences often assign “discussants” tasked with finding flaws in talks, and journals assign referees to criticize submitted papers.

Since this idea is so ancient, you might think that the people who talk the most about trying to overcoming bias would apply this principle far more often than do others. But from what I’ve seen, you’d be wrong.

Oh, almost everyone circulates drafts among close associates for friendly criticism. But that criticism is mostly directed toward avoiding looking bad when they present to a wider audience. Which isn’t at all the same as making sure they are right. That is, friendly local criticism isn’t usually directed at trying to show a wider audience flaws in your arguments. If your audience won’t notice a flaw, your friendly local critics have little incentive to point it out.

If your audience cared about flaws in your arguments, they’d prefer to hear you in a context where they can expect to hear motivated capable outside critics point out flaws. Not your close associates or friends, or people from shared institutions via which you could punish them for overly effective criticism. Then when the flaws your audience hears about are weak, they can have more confidence that your arguments are strong.

And if even if your audience only cared about the appearance of caring about flaws in your argument, they’d still want to hear you matched with apparently motivated capable critics. Or at least have their associates hear that such matching happens. Critics would likely be less motivated and capable in this case, but at least there’d be a fig leaf that looked like good outside critics matched with your presented arguments.

So when you see people presenting arguments without even a fig leaf of the appearance of outside critics being matched with presented arguments, you can reasonably conclude that this audience doesn’t really care much about appearing to care about hidden flaws in your argument. And if you are the one presenting arguments, and if you didn’t try to ensure available critics, then others can reasonably conclude that you don’t care much about persuading your audience that your argument lacks hidden flaws.

Now often this criticism approach is often muddled by the question of which kinds of critics are in fact motivated and capable. So often “critics” are used who don’t have in fact have much relevant expertise, or who have incentives that are opaque to the audience. And prediction markets can be seen as a robust solution to this problem. Every bet is an interaction between two sides who each implicitly criticize the other. Both are clearly motivated to be accurate, and have clear incentives to only participate if they are capable. Of course prediction market critics typically don’t give as much detail to explain the flaws they see. But they do make clear that they see a flaw.

GD Star Rating
Tagged as: , , ,
Trackback URL:
  • Philon

    “If your audience cared about flaws in your arguments, they’d prefer to hear you in a context where they can expect to hear motivated capable outside critics point out flaws.” Well, if your presentation
    is public–i.e., if it is *published*–and there is freedom of speech and the press, then you are *always* in such a context: if your arguments are worth criticizing, critics will present themselves.

    • Merely posting something on the web does not create a sufficiently motivated critic.

      • Philon

        If your post is sufficiently obscure–no, it doesn’t; if your post is sufficiently prominent–yes, it does.

      • Almost everything is “sufficiently obscure” in your sense.

      • Philon

        everything” is a considerable overstatement. The “marketplace of
        ideas”–especially the academic part of it–contains a vast amount of
        back-and-forth critical discussion.

      • rca32

        The lack of interest in critiquing an idea such as prediction markets could possibly be seen as ‘gateway’ criticism all on its own. Or to put it another way, the idea doesn’t have enough current potential status to be worth spending time critiquing in a detailed way. Piketty on the other hand is quite the opposite.

  • Philon

    For example, we didn’t have to wait long for the critics of Piketty to pile on. He didn’t have to solicit them!

  • Seems it might be more efficient to pay some conscientious and competent critics than to subsidize a prediction market.

  • Daniel Carrier

    Do you have any recommendations about where to find such motivated and capable critics?

    • I’ve had modest success using a combination of incentives and anonymity, though this is one of those “weird” things that most people just won’t take part of. Folks with higher status or wider readerships might get more responses than I have so far.

    • Motivation isn’t just found, it is also made. PAY them something.

      • rca32

        Paying critics may create an incentive structure that will work against your goal of getting improvement focused criticism as won’t the critic benefit in the longer term by being paid for criticism which doesn’t fix underlying problems with the whole idea but instead focuses on relatively inconsequential points in order to gain more income? Whereas a more savage critic will also be unlikely to generate much repeat business. Paying critics at the initial stages of an idea would also massively add to start up costs in terms of time and money and would probably be an innovation killer.

        I’m sure if an idea is strong/interesting enough to get some status (or potential status) it will find critics for free. If it’s not getting detailed criticism on its own merits (or lack of them) it’s probably got something major wrong with it to start with.

      • Daniel Carrier

        I’m cheap. Besides, I’m not someone who’s opinions will matter. This is basically something I want to do for fun.

  • Ryan Carey

    If you’re reading this and you’re not an effective altruist, it’d be good to hear your criticism:

    • People are hypocritical about charity. They say they want to help people, but they also care at least as much about other functions of charity. By exposing such hypocrisy, you might induce people to do things closer to what they say, but you might also harm the other functions, and make people less willing to do charity.

      • John_Maxwell_IV

        What are the other functions of charity?

      • The one which collides most directly with efficiency is (Veblenian) conspicuous waste.

      • John_Maxwell_IV

        If conspicuous *waste* was actually a goal, even inefficient charities are a relatively bad way to accomplish this. You could imagine burning a big pile of money, buying expensive cars and letting teenagers trash them, etc. There are plenty of more wasteful ways to spend money than giving to charity.

        It seems more likely to me that charitable giving is about demonstrating *wealth*, not waste. Giving $100K to an effective charity does this just as well to giving $100K to an ineffective one.

  • efalken

    The problem is, getting a reputation is necessary to get smart people to respond one way or another, and it’s hard to build a reputation without joining a side. Early on you build reputation within a small tribe that advances their parochial interest (say,libertarianism, socialism, the fruitfulness of dynamic programming or the efficiency wage theory), and by then you have the clout to get good feedback from influential people who invite you to conferences and assign reviews to predictably reliable referees. If you become an authority figure, your prejudices have probably become a habit, and seeking truth is less attractive than maintaining your status anyway.

    I think anyone seeking success in this world has to accept that one can’t say what they really think in public because it would put off too many potential allies. If you are lucky you have a small circle of friends where you can riff on such matters without worrying about damaging relationships AND getting good feedback (eg, my mom would love me no matter what, but her opinion on many issues wasn’t that valuable).

  • blink

    One reason to circulate among friends is that they are at least likely to *read* one’s paper. Critics are more likely to respond with silence, which reduces what may infer from hearing only weak criticisms.

    Second, a friendly audience is often the best place to start to flesh out big errors or erroneous details. Better to embarrass oneself in front of a small audience who will not give undue weight to a gaff than be written off as a lightweight by a serious opponent.


    I can see one advantage and one disadvantage for prediction markets in this matter. The advantage is that a prediction market is (relatively) anonymous and places a penalty on being wrong. This keeps away people who criticize for personal reasons or otherwise don’t really have a good reason to disagree. The disadvantage is exactly what you already wrote: there’s no explanation of why people agree or disagree with you, or even whether there is any consensus among your critics.

    The question is which of the two weighs the heaviest.

    You also won’t know if a lack of bettors means you’re right or if it means few people are interested in the market, but that same problem exists in other forms of fact checking (perhaps you’re paper is wrong but no one with the knowledge to criticize has read it) so I won’t count it as a disadvantage fkr prediction marmets.

  • John_Maxwell_IV

    So in concrete terms, what would this look like if implemented? For example, you allow comments on your blog. Does this constitute a sufficient “fig leaf”?

    I can think of a few ways a person or organization who wanted to enhance outside criticism could accomplish this:
    * Give outside critics a platform that’s equal to your own. For example, a blogger like you could have a policy that if a critical link got at least 4 upvotes in an open thread (disregarding downvotes), you would create a toplevel post sharing that link and responding to it if you wanted to.
    * Invite outside critics… brainstorm (and take suggestions) for people who might be capable critics, and then request that they criticize your stuff. Giving them a platform (as described above) could be an incentive for them to go through with criticism. Other incentives might be effective but you’d want to avoid destroying any intrinsic motivation on the part of the critics.

  • John Salvatier

    Is there a relatively easy way we could create prediction market contracts on a few EA related questions? I’d be willing to put some money into that.

  • Pingback: Recomendaciones | intelib()

  • Typo: extra if “And if even if your”