In my earlier posts in this thread I laid out a basic argument about what digital scholarship might be and might not be and how it could count in the tenure and promotion process in higher education. Today I’m going to discuss how we might get to a place where digital scholarship might begin to count.
In my second post I argued that scholarship has the following characteristics: it is the result of original research; it has an argument of some sort and that argument is situated in a preexisting conversation among scholars; it is public, it is peer reviewed; and it has an audience response.
I think that almost all historians would agree with this definition because it’s the one we use all the time. We’re comfortable with it, it works for us, and given how used we are to it, many historians (including many I know and respect) argue that there is no need to change it. After all, if it ain’t broke, why fix it? [For more on the evolution of peer review in the history business, see the American Historical Association’s statement on peer review.]
Alas for our current definition of scholarship, the digital world is undermining our certainties.
The big sticking point is the next to the last part of my definition–peer review. For a century or more, “peer review” in our discipline has meant that the historian produces his or her work (book/article) and submits it for publication. Then, after waiting months (or more likely many months) the historian finally receives feedback on his or her work and either has a little more work to do, a lot more work to do, or finds the work rejected entirely.
Why won’t this process survive in the digital world? The answer is pretty simple. It just takes too long and does not work in a medium where gatekeeping makes no sense. By it’s very nature, digital scholarship happens in a dynamic space–one where the work is often “self-published” in the sense that a scholar or a group of scholars creates historical work in the digital environment and then it is made available when it’s done (or close enough to done to show other people). Not after a lengthy process of peer review–but when it’s ready to be seen.
Then, and only then, does the peer review begin. The Internet is an open environment, not the closed environment of the publishing industry that we have lived with for many generations. Anyone can publish anything online and that, of course, means that a lot of dreck appears. But the fact that dreck is scattered all over the Internet does not mean that quality work cannot also be appear through the same process.
The AHA is proposing to try to act as some sort of gatekeeper for digital historical scholarship, but as I’ve already argued, this proposal is doomed because it is trying to find a way to fit the old system into a new technological environment where gatekeeping as we’ve known it doesn’t (and can’t) work.
Already in other industries we have seen what happens when the guardians of the old ways try to hold back the tide of change. Sales of music CDs continue to drop like a stone while sales of individual tracks through services like iTunes continue to rise rapidly. Two decades ago Kodak employed something like ten times as many workers as it does today (when was the last time you bought a roll of film?). And while Amazon.com hasn’t killed off all local bookstores, there certainly are fewer than there used to be.
So what, you might ask? Why do we have to change?
Because if we don’t, we’ll eventually become irrelevant. Already other disciplines that are not as resistant to change have embraced the digital world to a much greater extent. See, for instance, a recent article in the New York Times on the Social Science Research Network. Work posted on this network “counts” in many academic departments around the country despite the fact that peer review takes place after the fact, not before. And in other disciplines–computer science, biology, physics, etc.–peer review takes other forms entirely. So why are we so hung up on keeping a system that made good sense 100 or 50 years ago, but makes less and less sense today?
I wondered what a provost might think about this issue so I interviewed Peter Stearns, our Provost here at George Mason. For those of you who don’t know him, he’s a past vice president of the American Historical Association (Teaching Division), the founding editor of the Journal of Social History, and the author of more than 100 books. So he knows something about peer review.
What he told me was that being a Provost meant that he had to take a much more capacious view of peer review, because each discipline at the University has its own standards for what constitutes proper peer review. What Peter cares about is not how the peer review happens, but that it does happen. “It can be either before or after publication,” he said in our interview.
Other disciplines do it, so what is so particular, so unique about historical scholarship that it must be reviewed prior to publication? I can’t actually think of anything.
I’m not proposing that we throw out a system that has worked for so long in one fell swoop. But I am suggesting that there needs to be a serious discussion in our profession of what peer review means, what its value is to the process of advancing knowledge, and how it can change to take into account the new realities of the digital world. If we don’t have this discussion–and soon–we’re in danger of losing touch with a rising generation of young scholars who will see us as nothing more than cranky old scholars who are hanging onto an old system because it serves our interests and not theirs.
What do you mean when you say peer review happens after the fact?
Mills, your argument here is excellent. I agree with this part of your series wholeheartedly. Bravo.
Hey Mills.
In the current situation, perhaps the most important (visible) peer review happens after the fact, i.e. published reviews of monographs in particular. We can all think of really negative reviews, which suggest that pre-the-fact reviews are not sufficient by themselves (or can at least be undermined), and we know how important those reviews can be when it comes to promotion, tenure, and general scholarly reputation. So the current system has post-facto reviews already, although with minimal ability to modify the results (excepting response to the Editor, H-Net review discussions, second editions of works…).
Questions:
1) Would such online comments be included in one’s P&T application like book reviews are? I think I’d prefer a little more gate-keeping if my tenure application is on the line.
2) The main disadvantage with the status quo you mention is delay, which I agree with completely, but it seems that reviewing monographs is going to take a long time no matter what format it’s in. Are we likely to see even journal articles as digital scholarship, or something else shorter, more visual, etc.? OR, if the feedback will be immediate….
3) How useful is the immediate feedback that online posters provide? I find most conference paper presentations not very useful – rarely does the audience raise a point that the author hasn’t already thought of. And there aren’t that many academic blogs whose comments I find useful either, which makes me wonder about their utility. The medium seems to make them too disposable and ephemeral. My comments being the exception of course!
4) Would such a system encourage scholars to keep dwelling on their old research as new online comments appear, or would they go off on their next research project, as usually seems to be the case?
5) The AHA’s suggestion of playing online gatekeeper makes sense from the perspective of trying to judge the reputation of various websites (as we do with journals) – maybe journals could take charge of their own digital scholarship website, to play off their own reputation? Probably too expensive/too much work though.
Another short thought:
I’d bet a huge part of the time delay is due to personnel issues – either journal staffers are really busy (volunteers mostly I think), or the reviewers are volunteer academics and we know how long academics wait to return reviews (pre- or post-facto). So how exactly would digital scholarship shorten the time, while maintaining the ‘quality’ that presumably comes from thoughtful review, which is required for fairness to those up for P&T? What percentage of the delays come from these bottlenecks vs. space limitations and the publication process?
It is not clear to me how this essay fits into your larger series on making digtital scholarship count. The very interesting question of when and how peer review might change to fit the medium, seems very different from the question of why peer review may entrench biases against digital scholarship. Just because new media change the way we can do something, it does not necessarily follow that particular changes will happen when and as we expect. After more than a decade of frustrated expectations (I’m still waiting for paper use in my office to go down, instead of up), I can easily imagine that biases against digital scholarship could be carried over into post-publication peer review.
You write that “Work posted on [the Social Science Research Network] ‘counts’ in many academic departments around the country despite the fact that peer review takes place after the fact, not before.”
Actually, SSRN includes both “working papers” (not yet peer reviewed) and “accepted papers” (already peer reviewed, but awaiting publication), and even post-publication offprints. For my most recent article, the history journal asked that I not post a pre-review version, but allowed me to post a draft once we had gotten anonymous readers’ comments in the traditional manner.
I’d be interested to learn what disciplines count non-published working papers toward tenure and promotion, and how they value them. I know that in legal scholarship, SSRN is quite popular, but it is no substitute for publication in a prestigious law review.