Debugging Comments

Back in October, Greg Mankiw cancelled comments at his blog:

Unfortunately, a few (usually anonymous) commenters too often crossed the line.  I just don’t have the time to police comments and enforce good behavior, especially since some posts were generating more than 100 comments. And I don’t want to host a party in which a small vitriolic minority consistently tries to ruin the event for everyone else. So I decided to turn the comments feature off.

Xeni Jardin acted similarly:

When our audience was small in the early days, interacting was simple. … No moderation, no complication, come as you are, anonymity’s fine. … The audience grew. Fast. And with that, grew the number of antisocial actors, "drive-by trolls," people for whom dialogue wasn’t the point. … With much regret, we removed the comments feature entirely. …


We observed other big sites that included some kind of community forum or comments feature. Some relied on voting systems to "score" whether a comment is of value – this felt clinical, cold, like grading what a friend says to you in conversation. Dialogue shouldn’t be a beauty contest. Other sites used other automated systems to rank the relevance of a speech thread. None of this felt natural to us, or an effective way to prevent the toxic sludge buildup. …  Finally, this year, we resurrected comments on the blog, with the one thing that did feel natural. Human hands.  We hired a community manager. 

I feel Greg and Xeni’s pain; blog comments vary widely in quality, mainly because commentors vary widely.  But I am also deeply attached to the idea that intellectuals should open themselves up to challenge by a large able diverse community.  If you write something you want others to take seriously, readers should see that many diverse able people could challenge what you say, in a way that so that you would see their challenges and feel obligated to respond.  The lack of effective challenges can be your strongest endorsement. 

Unfortunately, for blog posts this community of challengers cannot usually be everyone who might want to comment on it.  If the author of a popular blog felt committed to give detailed responses to all challenging comments, the low average quality would produce a spectacular waste of his time. 

Of course we do now see bloggers who feel they should respond to challenges from similarly (or more) popular blogs.  But this seems far too few challengers to me.  So I have two half-baked ideas for allowing a wider circle of high-quality challengers:

  1. Define a set of elite commentors for your blog.  The blog author could flag some commentors as elite, either because he knows them from elsewhere, or because he was impressed by their previous comments.  Elite comments could be distinguished, such as by having their own section at the top, or a boldface heading.  If you saw many able diverse comments flagged as elite at a blog, you could believe the author had opened himself up to many challenges.
  2. Create a duel system where anyone could flag a specific claim in their comment as a "challenge", backed by some things they each valued.  If the author accepted the challenge, the two of them would conduct a debate, which then had some known (perhaps 10%) chance of being evaluated by some jury.  If the jury thought the challenger had sufficiently supported his challenge, the author would lose his valued items, while otherwise the challenger would lose his items.   

Of course Overcoming Bias won’t be implementing any such features unless someone  volunteers to create the software.

GD Star Rating
loading...
Tagged as: ,
Trackback URL:
  • http://www.ciphergoth.org/ Paul Crowley

    The work that doesn’t go into this astonishes me. The first blog I ever commented on was probably Slashdot. It had I think proper comment threding from the year dot, and since evolved registration, comment scoring, “karma” scoring of participants which in turn enabled comment scoring, meta-moderation, and a host of other measures to try and keep comments useful in the face of sophisticated attacks on their usefulness by unbelievable dickheads whose motivation still makes me wonder today.

    Slashdot’s system still has great flaws, of course. I’d assumed that future blogs would learn from Slashdot and this would continue to be an active area of research and learning. Instead practically everywhere I comment has taken great steps backwards, and mostly don’t even provide threading.

    For myself, I’m reasonably convinced that trust metrics are the Right Way to maintain comment usefulness in the face of spam and trolling. Your “elite commenter” proposal is a start along those lines, but in order to work, it would need to actively attract high quality dissenters; it’s my experience that (eg) left-wing blogs do not generally attract thoughtful right-wing commenters but only yahoos, and I wouldn’t be surprised if the same were true the other way around.

  • Caledonian

    And who decides what counts as high-quality? ‘Left-wing’ blogs are likely to reject ‘right-wing’ commenters regardless of how serious or respectful they are, and vice versa.

    Part of the trouble is that people are not willing to sit down and define what they mean by ‘troll’. Instead, they want it to include whoever and whatever they wish it to at the moment.

  • http://hanson.gmu.edu Robin Hanson

    Paul and Calendonian, let’s not let the best be the enemy of the good. Even if you chose elite commentors who tended to share your political leanings, you’d have still enlarged your set of challengers for most issues.

  • Rudi Schlatte

    Teresa Nielsen Hayden, who is boingboing’s community manager, has this to say about moderation and how to nurture a community where healthy online debate can occur:

    http://www.nielsenhayden.com/makinglight/archives/006036.html

  • http://www.midasoracle.org/ Chris Masse

    You might submit your ideas to the open-source software community. Somebody might code a comment plugin for WordPress or MovableType, with your specifications. Or maybe that plugin already exists.
    http://wordpress.org/extend/ideas/

  • http://michaelkenny.blogspot.com Mike Kenny

    Maybe a few simple rules for commenting would be good. I know some have just been created for overcomingbias.com. Perhaps a few more could be added along the lines of:

    1. Try to add to the conversation with facts or arguments or new suggestions or refinements. Avoid repetition, me-too-ing.

    2. Try to be friendly, especially when disagreeing, and when in doubt, assume someone criticizing your views is doing it in a friendly way.

    3. Avoid responding to a comments that don’t seem to conform to the above rules, in your judgment.

    Of course, maybe these are unnecessary (particularly here–I find the comments quite good generally myself), or different rules would be useful!

  • Ben Jones

    I nobly put my name forward for community manager of OB.com. It’s going to be a tough task but I feel that if I devote a couple of hours of each working day to keeping things civil and productive I will be performing a service to my fellow man that will make it all worthwhile.

    Seriously though, while Elite Posters would be a quick and satisfying fix, it would render a blog arguing for de-biasing humanity pretty toothless. Evaporate the proles and cool us all down!

    How about a heartfelt new year’s resolution from everybody that spends an appreciable amount of time here not to take the troll-bait, and to think ‘is this relevant, constructive and on-topic’ before hitting that ‘Post’ button?

    Or am I being irrationally naive and idealistic?

  • Caledonian

    Even if you chose elite commentors who tended to share your political leanings, you’d have still enlarged your set of challengers for most issues.

    It’s not the size, it’s the quality. Handpicking one’s “challengers” does not strike me as an effective way to challenge one’s thinking. By careful selection, you could even choose “challenges” that you were guaranteed to win.

    To put it another way: comments that you don’t like aren’t a bug. They’re a feature!

  • Dar McWheeler

    Hello,

    I’ve been enjoying your writtings for some time now. Kudos.

    As someone who’s been posting since the days of Fidonet BBS (yep, that old…lol) I agree with the whole comments problem. Many great discussions have had the legs knocked out from under them by people who, IMO, have anti-social traits. This low quality of input is the main reason why I comment so little these days. It’s also why I feel the need to say something possitive about your plight. You are a good writer and have some fine ideas.

    Keep up the great blog.

    Dar

  • josh

    You probably would lose a substantial portion of your audience if they were not allowed to comment, so the best course of action definitely depends on the intent of the author. I’m not sure how much a sermonizer like Eliezer (hey, that rhymes) would want to cut his audience down, even if it meant improving the general quality of the comments.

  • http://hanson.gmu.edu Robin Hanson

    Ben, Josh, and Cal, the elite commenter concept would not take anything away from ordinary commentors. As before, the poster could respond or not as he/she wished to ordinary comments. We would only add an expectation of a response to elite challenges.

  • Bob

    If I recall correctly, in Ender’s Game, Card had Net-based policy discussions organized in concentric circles. Everyone could comment in the outer circles on any topic (including those being discussed in the inner circles) but discussion within the inner circles was restricted using an undescribed merit system (you had to prove the worth of your ideas/thoughts/comments in the outer circles to become part of the inner discussion). I’ve always thought this would be a useful way to organize diffuse policy discussion but it would apply to a blogsphere rather than a single blog. Elite commenters is the same idea, though, but I don’t see why Robin would bother to designate them in advance unless the plan is to experiment with restrictions (e.g., only one comment per thread by ordinary commenters, with the possiblity that a good comment would lead to temporary or permanent elite status). Otherwise just let each comment be judged on its own merits and only respond to the worthwhile.

    In the meantime, I agree with Ben. Blog comments seem to stay civil until for some reason they fall into a death spiral sparked by a few bad apples. If we refuse to play, I see no reason why OB can’t avoid the problem.

  • Ed

    Why not have a rating system on posts (say -5 to +5) and then after a certain number of negative ratings the post gets deleted or archived. Community interaction and self policing does work and creates additional energy. It must be easy though.

  • Aaron M.

    Pre-post restrictions, such as limiting the number of comments allowed to non-elite posters, might act as a determent to people new to commenting or infrequent commenters. As suggested, the best solution is probably just to ignore trolling behavior, because trolling is an aggressive type of attention seeking.

  • http://dl4.jottit.com/ Richard Hollerith

    Here is a simple standard the community could impose on comments that does not require human discretion in its enforcement: require every comment to come with information on how to contact the commenter privately. Usenet had (and might still have) a standard in which the “From” field atop every post must contain a valid email address. The standard was not enforced by software, but almost everyone followed the standard till spam got really bad in the late 1990s and even then posters frequently complained about munged email addresses.

    Ben Jones, now that you’ve applied for a leadership position, shouldn’t your comments start to come with contact information?

    (To contact me, click my name, then click on the link named “contact”.)

  • Silas

    Robin_Hanson: Just as a little reminder, yesterday you said I made some good points, so, uh, just be sure to incorporate that when you consider your elite commenter system.

    That is all.

  • http://www.daublin.org Daublin

    By the way, two other related pieces of work would be Drexler’s “fact forums” and a defunct system I think was called crit.org. Both of these try to raise the best challenges to a post up to the surface. I would love to have access to that more easily. When I agree with something, I find that opponents often fail to engage at all what I think are the strongest arguments. When I disagree with something, I again often find myself unable to find good challengers amidst the storm of people attacking straw men.

    Robin’s proposal has the beauty of being easy to implement right now. By all means someone go for it. Please also try and convince Mankiw to do it, if anyone implements the software. His blog was better when people could talk back.

    As a refinement of the “elite commenter” status, the blogger could also have a “letters to the editor” queue from which the blogger would, at their discretion, read trhough and select things to post and occasionally mark a new elite commenter. Very popular blogs could go a step further and have elite “comment sifters” that go through the letters to the editor and pick some of the good ones to post.

  • http://profile.typekey.com/Unholysmoke/ Ben Jones

    Richard,

    Apologies – I have a Typekey account but don’t generally log in. Will do from now on. Please note the tongue in my last comment’s cheek.

    Robin – thanks for clarifying. Are you genuinely concerned that a number of commenters are detracting from the dialogue? I’m not enormously experienced in blog commenting, but I think that inevitable distractions aside, the average post here is pretty damn good. The ‘elite challenge’ idea does sound interesting, if a bit daunting. Rationality, however, tells me that if it’s likely to drive up the overall quality of comments, it must be a good thing!

  • http://www.satisfice.com/blog James Bach

    I get far fewer comments on my blog, but the system I use is to moderate and respond to each comment. If I think a comment is useless to anyone, I won’t let it post. If I think I disagree with a comment, but I think it raises an interesting issue, I’ll let it post but add my rebuttal.

    For you guys, I would suggest leaving things the way they are. Just ignore comments you don’t care about, and respond to the ones you do care about. People who comment don’t necessarily need you to respond.

    On the other hand, you could turn off all comments. That way people who want to comment simply do so on their own blogs.

  • Caledonian

    As before, the poster could respond or not as he/she wished to ordinary comments. We would only add an expectation of a response to elite challenges.

    So there isn’t an expectation of a response to any reasonable argument already? Hmmm…

    Rationality, however, tells me that if it’s likely to drive up the overall quality of comments, it must be a good thing!

    I think you need to listen to rationality more carefully. If the overall volume drops sufficiently, no increase in the overall quality will compensate. The point is not to ensure that only quality ideas are expressed, but that as many quality ideas as possible are expressed.

  • http://hanson.gmu.edu Robin Hanson

    Cal, no I do not feel an obligation to respond to any comment that might seem reasonable to an average reader. The average reader is in a poor position to judge such things, and such an obligation would produce a vast waste of time.

    Ben, yes like most blogs this blog has on occasion had low quality comments.

  • http://dl4.jottit.com/ Richard Hollerith

    A tip on the mechanics of providing contact information on this blog.

    I used to log in where it says, “If you have a TypeKey or TypePad account, please Sign in.” But then I learned it is much smoother (and avoids the empty-string-author bug) instead to fill out the form right below (Name, Email Address and URL) and check the box “Remember personal info”. Eliezer published the same tip a few weeks ago.

  • Mason

    I like the elite commentors idea, but I would favor puting them in bold in the thread, rather at the top, so that they can stay in the conversation, but their comments are still easy to find.

    I’d also like some sort of comment rating and/or category section, catagories like – challenges, supports, or irrelevant, and ratings -5 to +5 as previously suggested.

    Something that flags commentors who make x number of comments which recieved -5 / +5 rating may help you ban / promote more efficiently.

  • http://www.daegmorgan.net Raven Daegmorgan

    Seems to me many of these are attempts to solve what is essentially a human problem systemically/technologically, when such solutions are doomed to failure because they don’t actually solve the problem: like using flesh-colored paint to cover a wound, the color may change but the wound is still there. (I personally believe this occurs because many bloggers are technologically savvy, while at the same time often not as people/social savvy, and so their solutions to problems are based around technology while missing the human dynamics creating the problem in the first place.)

    I am a member of successful game/design-related community where the topic of moderation was recently discussed in an interview with the site maintainers/moderators, and a number of points pertinent to this discussion were made about the way people view and deal with the internet and others on the internet.

    A quick summary would be: most people are looking for emotional satisfaction — whether that be from safely getting away with uncouth behaviors, feeling like a part of a group, or gaining and maintaining a perception of status — in on-line discussions, leading to posturing, pointless arguing, and so forth. Combating this behavior requires developing and maintaining a community around standards accepted by the community, and then maintained/socially reinforced by the community, recognizing the difference between pointless discourse (as well as polite-but-pointless) and functional discourse, and not providing a community the environment in which emotional satisfaction-based posting would thrive.

    If you listen to the interview, keep in mind there are obviously game and design related issues discussed, but a decent chunk of the hour is about moderation and human behavior/interaction on the internet. The relevant discussion starts about 5 to 6 minutes in.

  • Psy-Kosh

    Maybe do something similar to the whole “list sniper”+open moderation thing of SL4, since it seems to kinda work?

    What I mean is, appoint some “blog snipers” that are given authority to moderate, delete comments, etc, but all such actions must be open and public? ie, make meta mosts along the lines of “moderator so and so has removed all of such-and-such’s comments from posting something-or-other because they were doing-something-that-they-were-doing”, but with the openness enforced by the software?

    That is, have what ammounts to a (publically viewable) log of all actions taken by the moderators, generated and updated automatically, such that it also allows one to view the “state of the site prior to the action”, analogous to being able to view the entire history of an entry on wikipedia.

    The official moderators perhaps should be mainly people other than the main posters of the blog, to help at least reduce any potential biases there.

    (note, to clarify, the publicising of the moderation actions ought to be both automated and manual. What I mean is there will be the publically viewable log/history system, plus also moderators should, after taking moderating actions, make some posting to describe what they did and why, with perhaps including links to the relevant parts of the moderation log)

  • rukidding

    It’s amazing how many people only believe in democracy and freedom of speech as long as they’re in the majority and are hearing what they want to hear.

    I have trouble understanding why people start crying because other posters are “insensitive.” It always means “disagrees with me.”

    Why do so many of you supposed intellects so greatly fear diversity of opinion? Why are you so sensitive to being disagreed with? And even in the case of truly mindless babble (at least as likely to be left-wing as right-wing), why the difficulty in just ignoring it?

    Comments are often more interesting that the posts. And they also serve the important function of showing how the posts are received. Even when I disagree with a commment, I’m still interested in learning about the mentality of those I disagree with.

    Then again, I’m not a sniveling little twit academic, terrified of the real world and harsh lighting, desperate to believe that “We’re right!” and “They’re wrong!”

    And your reaction to that last sentence is exactly my point. People who don’t like the comments section and fear diversity of opinion are like country-club whites talking about “those” people.

  • Leif

    I’ve never witnessed any major problems with comments on this blog, so I think a slashdotesque ranking system would be entirely too rococo for OB. Also, I like having all the comments on one page, so that you can ctrl+f certain terms or names easily.

    Besides, sometimes even the most boneheaded posts can provoke you to think, even if not in the direction intended by the author.

  • Caledonian

    Boneheaded comments, too.

  • http://bizop.ca michael webster

    I agree with some of the points made by Raven.

    Most academic blogs are mild compared to the hardcore flaming that goes on at various forums – but people survive.

    I also agree with Leif; boneheaded posts often make you think deeply about a problem.

    I don’t think the problem is really worth worrying about. But I could be wrong.

  • burger flipper

    In the spirit of the Edge.org project the Xeni quote was pulled from, I’d be curious to learn from Robin, EY, or another poster here if a comment has ever lead to a change of mind about what was posted?

    Is the model operating here really confrontational? I often see Robin critique EY’s posts, but don’t recall seeing too many resolutions in these cases (or even sufficient clarification on the premise to make rational disagreement impossible).

  • Al Cellier

    Brad DeLong, the fat Marxist blogger, censors all comments that fail to endorse left-wing political/economic stances. It’s scary that he stifles debate that might threaten his political biases.

  • Silas

    OT, but there hasn’t yet been an open thread for this month, so I thought I’d mention this here:

    Robin_Hanson’s idea for the bet about the repeated study on women vs. men talkativeness expired yesterday and it looks like the results confirmed the previous study. Comment?

    I can’t seem to link directly to the contract, as there’s some javascript predicate, but if you click soon you’ll get it here

  • Bob

    So, rukidding, you’ve either intentionally or unintentionally provided a great example of why limits on comments could be useful. Anything that increases the signal-to-noise ratio in the discussion is worth considering.

  • Caledonian

    Feel free to offer explicit definitions of signal and noise – ones that don’t allow you to shove positions you don’t agree with into the category of ‘noise’.

    Given the breathtaking hubris and shoddy argumentation present here on a regular basis, statements like rukidding’s are desperately needed. If nothing else, they permit a demonstration that contrary positions aren’t run off the blog.

  • http://www.fitzblog.com Patrick Fitzsimmons

    You guys should consider using Disqus on your blog ( http://www.disqus.com ). It allows you to set up a voting system that will automatically promote the good content to the top and filter out the junk.

  • Noumenon

    Slashdot’s comment rating system is awesome. YouTube’s comment section would be about ten times as useful if their new comment rating system would let you screen out comments at level 1, 2, 3, or 5 like Slashdot.

  • Gray Area

    ‘Elite tags’ don’t make the ‘mafia effect’ better, and certainly give the appearance of making it worse.

    Low quality posts get ignored. Consistently low quality posters get ignored consistently. I think it is better (for everyone, really) to slog through content the hard way to determine high quality posts than rely on a sign system which declares ‘THIS IS A HIGH QUALITY POST.’