Academic Blog Credit?

Martin Weller considers academic credit for blogs:

The answer to … whether new approaches such as blogging constitute scholarly activity, is an emphatic yes. Which leads us to a more problematic question: How should we recognize it? …

Tenure committees have increasingly come to rely upon journal-impact factors to act as a proxy for research quality. In short, we know what a good publication record looks like. But these criteria begin to creak and groan when we apply them to blogs and other online media. Simple metrics are subject to gaming, and because of the removal of the peer-review filter, may be meaningless anyway. I may have a YouTube clip of a skateboarding octopus with two million hits, but that doesn’t make it scholarly work.

It’s a difficult problem, but one that many institutions are beginning to come to terms with. Combining the rich data available online that can reveal a scholar’s impact with forms of peer assessment gives an indication of reputation. Universities know this is a game they need to play—that having a good online reputation is more important in recruiting students than a glossy prospectus. And groups that sponsor research are after good online impact as well as presentations at conferences and journal papers.

… I’ve found that since becoming a blogger, I publish fewer journal articles, so it has had a “negative” impact on that aspect of my academic life. However, it has led to so many other unpredictable benefits—such as the establishment of a global peer network that helps me stay up to date with my topic, increased research collaboration, and more invitations to give talks—that it’s been worth the trade-off. (more)

Yes universities care about getting good and much “press”, but they are not willing to tenure professors merely for getting good press. The self-concept of professors only lets them give at most a minor weight to press, and sometimes the weight is negative.

The key difference is between getting attention vs. making impressive original intellectual contributions. Being cited by major news media, or having so many blog readers, can credential you as getting attention. But so far only journal articles, Ph.D. theses, and certain books and conference papers are accepted as credentials for impressive original intellectual contributions. For these, high quality experts are seen to judge the intellectual contribution.

Yes blog posts can contain impressive original intellectual contributions. Newspaper columns can contain them as well. So can speeches. Even spontaneous party conversations can contain them. The problem is, we don’t have systems set up for experts to evaluate these things in such terms. And if an intellectual contribution isn’t credentialed as such by academic experts, then it basically doesn’t exist as far as academia is concerned.

So either blogs will be continue to be seen mainly as a way to get “press” attention, or some folks will develop a system of expert evaluation of the intellectual contribution of blog posts. And as with academic journals, the main obstacle to doing that is: getting sufficiently prestigious academies to spend enough time doing their evaluations.

Now it turns out that many prestigious academic already read a lot of blog posts. So one approach would be to create a special “review” section where only prestigious academics can enter quick reviews of blog posts they read. Perhaps these reviews would be anonymous to the blog author and readers, and a more centralized part of the system would weigh their prestige, and degree of topical expertise, to compute a post evaluation.

But even with lots of new whizzy software support, it isn’t obvious you’d get enough reviews to make it work. People write reviews of journal articles in part because they hope doing so will favorably dispose editors toward their later submissions. People who write reviews of blog posts couldn’t have that motivation.

GD Star Rating
loading...
Tagged as: ,
Trackback URL:
  • http://www.uweb.ucsb.edu/~criedel/ Jess Riedel

    People write reviews of journal articles in part because they hope doing so will favorably dispose editors toward their later submissions. People who write reviews of blog posts couldn’t have that motivation.

    The natural Robin-Hanson response: why not just pay them? The convention of journal referees working pro-bono is a somewhat stable equilibrium (although the current trend toward making publications freely accessible puts that slightly at risk), but the development of a brand new blog-credentialing industry would give plenty of opportunities for a new norm to develop.

  • Vaniver

    Why not just use something based off Reddit like lesswrong, but require .edu emails to make accounts? You’d probably also want to differentiate between professors, grad students, and undergrads- you could even go so far as to not let the undergrads in. Their votes could be tallied separately, or given different point values.

    You could make the voting entirely separate from the blog-hosting, for universities that want to keep things at their domain / make it so established sites like overcomingbias don’t have to move.

    It seems like karma would have a few benefits over citations. It would make it so negative results get more acknowledgement- right now, if a paper saves me three months of wasted effort by diverting me away from a minefield, I can’t pay the author back by citing them if I move in a completely different direction, but I would definitely upvote such a post. As well, currently some citations count that really shouldn’t- if I write a paper tearing yours apart, your citation index goes up. If you publish a retraction, your citation index goes up.

    Moving science to blogs also makes it move faster, and makes it so outside (but relevant) experts can more easily have a voice. A statistics professor with a bone to pick could go through posts, downvoting any that have statistics errors.

    Speaking of relevance, you may be able to cut down on trolling by having everyone’s name attached to both their account and their votes. Since reputation is worth a lot in academia, staking yours on, say, karmassassination seems like a terrible idea.

  • http://don.geddis.org/ Don Geddis

    It seems like the current system already basically works. It is costly for high-quality academics to read, understand, and review a new piece of work. Submitting a carefully written paper to a journal process, is the signal that you wish this particular piece of work to be subjected to the expensive evaluation process.

    Blogs provide press, yes. They may even be the source of grand new ideas. But you don’t need to evaluate the posts themselves. When the grand ideas come, the blog author can repackage them as a journal article, which is the sign to society at large that this particular idea is worthy of being judged by the highest standards.

    I don’t see a problem that needs solving here. It seems vastly more efficient for the authors to choose themselves which of their ideas is worthy of being considered, rather than trying to create a new system to evaluate all blog posts, just on the off chance that some of them contain great ideas — ideas which the author himself either overlooked, or didn’t have time to argue under the current journal system.

    • http://www.uweb.ucsb.edu/~criedel/ Jess Riedel

      In addition to what Robin said (“we might want to get many more small insights, of the size that fit in a blog post, rather than fewer bigger ones, of the size that fits in a journal article.”) there is also the problem of turn-around time. Richard Price, the CEO of Academia.edu, wrote an editorial at TechCrunch the other day which expands on these ideas. Obviously, he’s not an impartial analyst.

  • http://hanson.gmu.edu Robin Hanson

    Jess, where would the revenue come from to pay for that? Academics would frown on blog authors paying reviewers to rate their blogs.

    Vaniver, merely knowing that someone was a professor someone in something would not be remotely enough topic-specific expertise.

    Don, the point is that we might want to get many more small insights, of the size that fit in a blog post, rather than fewer bigger ones, of the size that fits in a journal article.

    • http://www.uweb.ucsb.edu/~criedel/ Jess Riedel

      Blog authors would pay a fee to a blog credentialing service (analogous to publication fees paid to journals), which would hire an independent editor to seek out and pay other academics to rate the blog. The blog would get a shiny official rating sticker from the Prestigious Blog Review.

    • Vaniver

      Vaniver, merely knowing that someone was a professor someone in something would not be remotely enough topic-specific expertise.

      How specific are citation indices? I thought it was the total number of times you had been cited in any journal, not just specific journals. I agree that it takes more effort to publish a paper that cites an unrelated paper than it is to Like a post I have no expertise about, but this is again something you can track and discourage.

      • http://www.edwardrcarr.com Ed Carr

        Even more to the point, citation indices do not discriminate between positive and negative citation – that is, if you can write an absolutely terrible article and get it published, you could have hundreds of citations to it as everyone points it out as an example of what not to do – but that is still a lot of citations!

    • http://daedalus2u.blogspot.com/ daedalus2u

      How about the same way that Credit Ratings Agencies get paid. How do S&P and Moody’s get paid? Who pays to have investments rated?

      We have seen how well that system works. If it is good enough to rate trillions of dollars of assets, surely it must be good enough to rate science papers.

  • Rodney

    By this standard, Roissy, Yudkovsky, Kanazawa and many others should be offered prestigious chairs and Robin Hanson should be knighted and given a nobel prize. This is hillarious but for the scary thought that this might actually come to pass.

    • Salem

      Inevitably our views of this are coloured by our views of existing academia and its social value.

      I do find it interesting that RH leaves out an alternative; that academic status becomes less important.

    • http://eradica.wordpress.com Firepower

      The Romans had a useful tradition, to keep pride tamed, during victory parades.

      One must ALWAYS be careful not to over-credit someone simply because of longevity, seniority or popularity: that, is called American Politics.

      We see, how that functions.

  • PG

    Why not create a PageRank-like system where each author’s rank = blog author’s citation count + f(followers’ rank), recursively. Initial score is then built up through peer-reviewed journal rankings, but newcomers without a citation count can get rank by getting enough academic followers. f needs to be constructed so that it gradually takes over from citation count as the determinant of relative rankings.

  • http://omicron-theta.blogspot.com/ Ari

    Robin, if you look at Stack Overflow you will see how programmers help others, and get rewards. Some of this is just psychological trickery aka gamification. I think friend of mine does academic research on that. Believe it or not, but some companies are interested in people’s Stackoverflow reputation (to see what kind of programming problems they can solve). Github is more popular and doesn’t have any credits, but your code repositories and commits. Newer companies are interested in that, in fact I got asked about it in my last interview. If you browse both sites you’ll quickly learn their idea.

    I think with bit of designing and coding you could possibly set up some kind of system where you get virtual points, awards or badgets by fellow academics for producing great blog posts. Maybe a website where your blog posts will be automatically submitted and categorized depending on criteria. Basically it could be kind of a blog reader. Then other academics can give points of your from there or from your website. The hard problem will be convincing people the costs outweigh the benefits. You need certain amount of critical mass to get it moving.

  • Bo

    In my information science masters program, we had an assignment that tackled this very topic. Basically, we compared the results of google scholar’s ranking algorithm to the impact factors (or other available bibliometric data) from established, credential laden, sources (web of science, etc.) It still requires a good deal of manual effort (eg google reported the same article, published in different journals, as separate), but the basic method of using web analytics on academic content works rather well.

    Semantic web technology may one day be able to do all of the work, but in the interim, I agree with the comments: I like a part computational/part human review, web based system.

  • http://richardhserlin.blogspot.com/ Richard H. Serlin

    In academia currently certain journals have come to be considered more prestigious than others, and pubs in those journals carry all the weight for reward and punishment.

    In academia, certain blogs are already considered more prestigious than others, so quality/contribution could be rated based on posts on your posts in top blogs, as well as quotes from your posts, and links to them.

    Some respected academics could even rank blogs in top journal articles, say A through E, and you could calculate statistics based on that.

    • http://richardhserlin.blogspot.com/ Richard H. Serlin

      The peer review comes very directly from top bloggers like Paul Krugman, Simon Wren-Lewis, yourself, etc. deciding to quote your post, or post on it, or link to it.

      • http://hanson.gmu.edu Robin Hanson

        That is attention, but not evaluation for an original intellectual contribution.

      • http://richardhserlin.blogspot.com/ Richard H. Serlin

        I’m not sure the level of originality you have in mind (unless it’s 100% plagiarized, it’s original to some extent).

        I thought of this because I have little prestige; I never finished my dissertation. So when Brad DeLong or Rajiv Sethi blog about a post or comment of mine, it’s not because I’m famous; it’s only because they think it’s good in at least some important way (or especially bad, but that’s easily determined).

        In the case where someone is already prestigious, or famous, then all the citing in the quality blogosphere could just be because someone prestigious or famous said it, but in that case academic evaluation is not as important.

        If someone doesn’t have tenure, is not famous, and especially if they’re not at a top school, but is still getting lots of positive citing from top blogs, that probably says something good that’s not that hard to see and quantify. And for such a person, having that evaluated in academic reward is important.

  • http://richardhserlin.blogspot.com/ Richard H. Serlin

    Of course, it could just be that the blogger’s a good writer, a good teacher, a good communicator, of underlying ideas that aren’t original, and that’s why they’re getting all of the cites in the quality blogosphere. But would you nonetheless want to reward that in academia?

  • http://richardhserlin.blogspot.com/ Richard H. Serlin

    Or they could just be good, in a given post, at understanding and applying to the current problems and arguments ideas that aren’t original. But again, would you nonetheless want to reward that in academia?

  • http://richardhserlin.blogspot.com/ Richard H. Serlin

    Again, there’s probably something good (or especially bad) they’re doing if they’re being cited by quality people, and it’s not due to being prestigious or famous already.

    But yeah, there’s some vagueness, unless you evaluate the content of their blog directly and not with just simple “objective” criteria. But this happens to a large extent anyway in separating an average AER pub from a very important or seminal one, the pub is read and evaluated directly, and not judged just on simple “objective” criteria for its quality.

  • Pingback: The Success of Stack Exchange: Crowdsourcing + Reputation Systems « Permutations

  • http://richardhserlin.blogspot.com/ Richard H. Serlin

    I guess an additional idea is just paying for evaluation of your blog. I think I’ve heard of journal submission fees in the hundreds of dollars. You’re essentially paying for respected evaluation of the quality of your paper.

    What if you could pay a thousand, or a few thousand, for an evaluation of the current quality of your blog, A through E, and no ranking, from the AEA, or just AEA certified or not AEA certified, or another level AEA certified top blog, say? For those with good blogs it might be worth paying even $5,000 or more, and if the blog is good enough it might be worth it for the department to pay.

  • http://quomodocumque.wordpress.com JSE

    Robin, you might be interested to know that there is an explicit mechanism for academic recognition of blogging, at least in the hard sciences. All of us whose research is funded by the National Science Foundation have to include a discussion of “broader impact” in our grant proposals. And being able to show that you have a widely read blog on which science takes place counts for this, and helps you get funded.

  • Emil

    The solution is already in place and widely used. You use it every time you blog about someone else’s blog post. You are actually reviewing it openly and allow further interaction and improvement. Your blogging platform gives you identity (important so no one can forge a review) and also the technical means to track references (trackbacks and such). The only thing to do is to have a system that navigates the web of reports, comments, links and update a score every time an article is identified on someone’s blog that is known as a trusted reviewer. Efforts are underway to completely automate this process.

  • Michael Wengler

    Tangential to the consideration of how to add blog posts to the tenure/promotion calculation are some thoughts on the deeper changes in the world and how they shift the whole academic equation.  

    For the longest time, books, paper, literacy, and so essentially information of any quality was in scarce supply.  In such a world access to these things becomes highly competitive, and such is a good thing: if something is scarce you want a significant fraction of the “cost” of the thing being instrumental in finding those who will create the most with those scarce resources.  So the old academy is like a monastery with incredible trappings of rank and significant barriers to entry for the small number of spots available.  

    Long before the internet, publishing and transportation technology and the great overall increase in society’s wealth shifted those equations a lot.  When I was a grad student then professor in the 1980s and 1990s, I needed to publish in journals for the status.  But almost everything I wrote was read by the largish community who mattered long before it “appeared in print.”  This through first copying machines and cheap mail, and later through laser printers and email.  

    There is now a gigantic continuum of “knowledge workers” from ivy league professors through talking-head punditocracy, down through infotainment networks and out in to every other conceivable corner of the blogosphere.  This reflects a proper use of the conversion of the resource of information from something scarce and expensive to something whose cost of acquisition and access is trivially small compared to the personal costs of handling the information you choose not to ignore.  Information got “too cheap to meter.”  

    So yes, it is fun to consider how an ancient almost monastic institution of the academy, so formed by the high relative cost of information, “should” evolve in a world of free information.  I would love to see more discussion of what the features of modern institutions in a world of “too cheap to meter” information will be the efficient ones, whether or not it is the great-great-grandchildren of the original academies who manage to accrete to themselves those features, or whether it is unrelated institutions that do it.  

  • Pingback: What’s the problem? |

  • Pingback: Overcoming Bias : Fixing Academia Via Prediction Markets