Saturday, March 17, 2007

Quantifying Scholarship

A controversy emerged earlier this week regarding the use of the F-word in an academic paper. Actually, the controversy about the paper itself played out some time back; but the new controversy was over whether a provocatively-titled paper, "Fuck," by Ohio State law professor Christopher Fairman, 28 Cardozo Law Review 1171 (2007), should be disqualified when counting the number of downloads for which a law school should be credited. Say what?

The Social Science Research Network (SSRN) is a central repository into which scholars in many fields place their written work, where other scholars can then easily search for working papers on particular subjects and download papers that are interesting and/or potentially useful in one's own scholarly work. This provides a nearly ideal internet-era medium through which scholars can interact with each other at the draft stage of their writing, soliciting feedback from both supportive and skeptical critics and basically enhancing the scholarly process in every way. I don't mean to be glib. This is truly a wonderful resource.

The problem comes from our seemingly irresistible desire to count and rank things. (Rankings can be fun, of course. See "High Fidelity" for a wonderful portrait of dysfuntional men who have a "Top 5 All Time" list for everything: breakups with girlfriends, songs about death, etc.) SSRN downloads have become a method by which individual scholars and their employers are evaluated: a large number of downloads of one's papers moves a person up the rankings, and the presence of many heavily-downloaded professors on a law school's faculty moves that law school up in such rankings. The theory is evidently that multiple downloads imply "importance" or "influence" or something that makes a scholar appear to be having a useful professional impact.

One such use of SSRN to rank faculties was Brian Leiter's "Most Downloaded Law Faculties, 2006," posted on March 6 of this year, in which Leiter compiled rankings of the top-15 law faculties based on total downloads of all papers by all of a school's faculty and on downloads per paper of all of a law school's faculty. The immediate controversy has revolved around Leiter's decision to exclude Ohio State and Emory from the rankings because their high download numbers are overwhelmingly the result of Fairman's one famous paper. (Fairman is visiting at Emory.) Paul Caron reports on the controversy on the TaxProf blog here. For what it's worth, I agree with Ann Bartow, who criticized Leiter's decision on the Feminist Law Professors blog. She points out, among other things, that Fairman's paper is a very serious piece of scholarship about the impact of language on society, noting the irony that Leiter's decision reflects one of the very issues that Fairman discusses. Fairman has also posted a commentary about the controversy, which one can download from SSRN. (I love irony.)

The more interesting issue to me, though, is the notion of using SSRN downloads as a measure of scholarship in the first place. Others have offered varying critiques, including Leiter himself, who has pointed out that SSRN tends to be used more by scholars writing in certain areas of law than others and suggesting various ways in which the results can be gamed or at least skewed. All true, I have no doubt. Even more fundamentally, the problem with download counts as a measure of importance (or quality or whatever) is essentially identical to the older practice of counting citations to assess scholarly impact. If Mike Dorf, say, writes an article that everyone thereafter cites, that must mean it's important, right? Mike (whose articles are heavily cited, for all the right reasons) will be the first to tell you that that is simply not a valid conclusion. I have seen plenty of papers in economics and in law that are downloaded heavily apparently because they're so silly or so dangerous that they need to be debunked and exposed to ridicule. That's one type of "importance," I suppose, but it seems odd to give someone credit for saying something that other people find ridiculous or scary. It is odder still because the only people who are likely to be able to say something silly yet still be cited rather than simply ignored and dismissed are scholars who already have big reputations. So a big name at a big school can enhance one's scholarly reputation by being cited (and now downloaded) for writing something absurd. Put more simply, quantity measures are not quality measures.

There is an important difference, though, between how citation counting distorts behavior and how SSRN download counts distort behavior. Citation counting has almost surely led scholars to cite their own papers and their friends' papers extensively (even when those papers are not on point), to refuse to cite high quality articles with which they disagree, etc. That is a bad result, and it makes the citation-based rankings even less well-suited to measure importance or quality than they would otherwise be.

For SSRN, though, this distortion is especially poignant. When people browse SSRN, they now know that every decision to download a paper is a vote. If I see an abstract for a paper that might or might not make a ridiculous point, for example, I would normally want to download the paper to see what the author is really saying. If the author seems to be writing in good faith but is making a point poorly or has apparently not thought about some implications of an argument, I might even try to contact the professor to discuss the paper. I hope that others will do the same for me. If I do download a paper, though, I'm voting blind. I may find myself enhancing the measured importance of a paper that turns out not to be a worthy piece of scholarship (or even coherent). Decisions to download thus become strategic, discouraging scholars from even engaging with one another because of each scholar's subjective decisions about who and what one wants to support.

And that is especially troubling, because the promise of SSRN was that it would allow scholars to interact at the stage where their work would benefit from the views of others. Counting downloads from SSRN has led to rankings based on those counts, and that undermines -- perhaps fatally -- SSRN's greatest contibution to the academic enterprise.


egarber said...

Interesting. Are the downloads free?

So it's the age-old question of how to measure quality.

I could write a paper that offers little new, but plays to a timely issue like habeas corpus.And maybe that drives hundreds of downloads. (Ok, maybe not me :) ).

But another guy could write the best paper of all time on the third amendment, and he might never get downloaded.

Which example is of higher "quality"?

Neil H. Buchanan said...

Yes, the downloads are free. In addition to the choice of sexy versus obscure topics, there are also all kinds of ways to manipulate the process. For example, an acquaintance from another law school called me recently and asked if I would please download all of his papers from SSRN. His dean is making a big push to pump up their numbers.

To SSRN's credit, they do prevent multiple downloads from the same IP (or email) address, but there are plenty of other ways to pump up the numbers. Professors who teach large courses, for example, can assign their papers and have a hundred downloads in one shot. That's enough to show up on a weekly most-downloaded list. Which, of course, makes the professor an Important Scholar.

In some ways, the amazing thing is that this hasn't already gotten even more out of hand.

Anonymous said...

免費A片, ut聊天室, AV女優, 美女視訊, 免費成人影片, 成人論壇, 情色交友, 免費AV, 線上a片, 日本美女寫真集, 同志聊天室, 聊天室交友, 成人文章, 成人圖片區, 色情網站, 辣妹視訊, 美女交友, 微風成人區, 色美媚部落格, 色情影片, 成人影片, 成人網站, 免費A片, 上班族聊天室, A片,H漫, 18成人, a漫, av dvd, 一夜情聊天室, 微風成人, 成人圖片, 成人漫畫, 情色網, 日本A片, 免費A片下載, 性愛, 成人交友, 嘟嘟成人網, 嘟嘟成人網, 成人貼圖, 成人電影, 成人, 中部人聊天室, 080中部人聊天室, 成人貼圖, 成人小說, 成人文章, 成人圖片區, 免費成人影片, 成人遊戲, 微風成人, 愛情公寓, 成人電影, A片, 情色, 情色貼圖, 情色文學, 做愛, 成人遊戲, 成人影城, 色情聊天室, 色情小說, 一葉情貼圖片區, 情色小說, 色情, 寄情築園小遊戲, 色情遊戲, 成人網站, 麗的色遊戲, 色情網站, 成人論壇, 情色視訊, 情色電影, aio交友愛情館, 言情小說, 愛情小說, 色情A片, 情色論壇, 自拍, 癡漢, , 俱樂部, 豆豆聊天室, 聊天室, 色情影片, 視訊聊天室, 免費視訊聊天, 免費視訊, 視訊交友90739 情人視訊網影音視訊聊天室 免費視訊聊天室 視訊聊天 視訊交友 美女視訊 視訊美女 視訊 免費視訊 免費視訊聊天 視訊聊天室 辣妹視訊 一夜情 色情a片 aio交友愛情館 情色電影 情色視訊 色情遊戲 色情 情色小說 一葉情貼圖片區 色情小說 色情聊天室 情色交友 成人論壇 成人網站 色情網站 情色論壇 小高聊天室 女同志聊天室 6K聊天室 080苗栗人聊天室 080聊天室 聊天室尋夢園 UT男同志聊天室 男同志聊天室 尋夢園聊天室 UT聊天室 聊天室 豆豆聊天室 A片 成人電影 成人貼圖 嘟嘟成人網 美女交友 本土自拍 成人交友 成人影片

Anonymous said...

It is also can not live without eve isk. Here, for the violence or robbery had been friends apologize in the eve online isk. I request members are on the local increase or decrease in the number of judgments quickly in buy isk. The game has a good excellence is that the cheap eve isk. The other side rather than back into the space station is POS, and buy eve online isk.

Anonymous said...

酒店喝酒,禮服店,酒店小姐,制服店,便服店,鋼琴酒吧,兼差,酒店兼差,酒店打工,伴唱小姐,暑假打工,酒店上班,日式酒店,ktv酒店,酒店,酒店公關,酒店小姐,酒店兼差,酒店上班,酒店打工,禮服酒店,禮服店,酒店小姐,酒店兼差,寒暑假打工,酒店小姐,台北酒店,禮服店 ,酒店小姐,酒店經紀,酒店兼差,寒暑假打工,酒店小姐,台北酒店,禮服店 ,酒店小姐,酒店經紀,酒店兼差,寒暑假打工,酒店小姐,台北酒店,禮服店 ,酒店小姐,酒店經紀,酒店兼差,寒暑假打工,台北酒店,禮服店 ,酒店小姐,酒店經紀,酒店兼差,寒暑假打工,酒店小姐,台北酒店,禮服店 ,酒店小姐,酒店兼差,寒暑假打工,酒店小姐,台北酒店,禮服店 ,酒店小姐,酒店經紀,酒店兼差,寒暑假打工,酒店小姐,台北酒店,禮服店 ,酒店小姐,酒店經紀,酒店兼差,寒暑假打工,酒店小姐,台北酒店,禮服店 ,酒店小姐,酒店經紀,酒店兼差,寒暑假打工,酒店小姐,台北酒店,禮服店 ,酒店小姐,酒店經紀,酒店兼差,寒暑假打工,酒店小姐,禮服店 ,酒店小姐,酒店經紀,酒店兼差,寒暑假打工,酒店小姐,禮服店 ,酒店小姐,酒店經紀,酒店兼差,寒暑假打工,酒店小姐,禮服店 ,酒店小姐,酒店經紀,酒店兼差,寒暑假打工,酒店小姐,禮服店 ,酒店小姐,酒店經紀,酒店兼差,寒暑假打工,酒店小姐,禮服店 ,酒店小姐,酒店經紀,酒店兼差,寒暑假打工,酒店小姐,經紀 彩色爆米花,經紀人 彩色爆米花,酒店傳播,酒店經紀 彩色爆米花,爆米花,童裝,童裝拍賣,童裝大盤,童裝寄賣,童裝批貨,酒店,酒店,童裝切貨,酒店,GAP童裝,酒店,酒店 ,禮服店 , 酒店小姐,酒店經紀,酒店兼差,寒暑假打工,招待所,酒店小姐,酒店兼差,寒暑假打工,酒店上班,暑假打工,酒店公關,酒店兼職,酒店經紀

Anonymous said... . .
[url=]puma shoes[/url]
[url=]chaussures puma[/url]
[url=]nike air max ltd[/url]

Anonymous said... . .
[url=]puma shoes[/url]
[url=]chaussures puma[/url]
[url=]nike air max ltd[/url]

Anonymous said...

酒店經紀人, 菲梵酒店經紀, 酒店經紀, 禮服酒店上班, 酒店小姐兼職, 便服酒店經紀, 酒店打工經紀, 制服酒店工作, 專業酒店經紀, 合法酒店經紀, 酒店暑假打工, 酒店寒假打工, 酒店經紀人, 菲梵酒店經紀, 酒店經紀, 禮服酒店上班, 酒店經紀人, 菲梵酒店經紀, 酒店經紀, 禮服酒店上班, 酒店小姐兼職, 便服酒店工作, 酒店打工經紀, 制服酒店經紀, 專業酒店經紀, 合法酒店經紀, 酒店暑假打工, 酒店寒假打工, 酒店經紀人, 菲梵酒店經紀, 酒店經紀, 禮服酒店上班, 酒店小姐兼職, 便服酒店工作, 酒店打工經紀, 制服酒店經紀,,

Unknown said...

Those are best online website , best service , best quality. Good luck !
ed hardy clothing
Chaussures Sport
Tennis Racquet Shop
Cheap Polo Shirts
The North Face Jackets
cheap ed hardy
Chaussures Sport
Tennis Racquet
nike shox r4
ed hardy
cheap ed hardy
polo shirts
cheap polo
Remise Chaussures Sport
nike tn requin
ed hardy clothes
nike femmes chaussures