Making Digital Scholarship Count

In June 2008, I wrote a series of three posts, gathered here in one extended text, that I called “Making Digital Scholarship Count.” At the time, I was a full year past tenure and promotion to the rank of Associate Professor, and I had included digital work in my case for promotion and tenure here at George Mason University.  Those reviewing my case for tenure and promotion took this work into account, although more attention was given to my analog scholarship as being essential to my case. At the time I wrote the series, I felt that as a profession, we were at the front end of a transition, the end point of which would be a place where digital work and analog work would be assessed based on their scholarly merits, using similar but also different criteria, each appropriate to the genre of the work. As I write these works five years later, we’re still in the very early stages of that transition—at least in history—and I’d be hard put to predict when the situation I advocated for in 2008 will come to pass. In May 2013, I was promoted to full professor, at least in part on the strength of further digital work, as well as another analog book, this one about teaching.

I have created a small archive of the various documents relevant to my two cases for promotion in the hopes that they might help others building their scholarly portfolios with at least some digital work as part of that portfolio. A further discussion of the issues raised in the essay below took place on the Digital Campus podcast in July 2008. My colleague, Sean Takats, has also written more recently on this issue in his blog:  A Digital Humanities Tenure Case (1), and, A Digital Humanities Tenure Case (2). Finally, the Journal of Digital Humanities recently explored this issue from a variety of angles. Taken together, these various resources should provide at least some helpful guidance to those seeking to build their careers around digital work in the academy.

Making Digital Scholarship Count
June 2008

Today I am inaugurating an extended series of posts on the question of how digital scholarship should “count” in the ways that things count at colleges and universities. As more and more scholars do work in the digital environment they are expecting this work to count toward tenure, promotion, and other types of formal evaluation (in the hiring process, for instance). It is because I have attended numerous conference sessions and meetings over the past several years where this topic has come up (often in very passionate ways), that I have decided to write this series.

The other motivation for the series was an email I received from a friend and colleague whom I respect very much taking issue with something I wrote after my experiences at our THATCamp unconference here at CHNM. In that post I said that one reason why historians doing digital work find it difficult to receive full credit for their digital work (as opposed to other things they do like publishing a book or an article, teaching a class, or providing service to their institution) is that historians are a fussy and conservative tribe generally resistant to innovation.

My friend’s email suggested that this sort of name calling (his formulation, not mine) would only set back efforts to convince historians to accept digital work as “scholarship” on a par with other forms of scholarly work. I’m standing by my characterization of my tribe as fussy and conservative when it comes to innovations–after all, it’s worth remembering how hard it was for women’s history or cultural history to be accepted as a valid approaches to historical scholarship.

But, having said that, I do think that something needs to be done–something other than complaining about my colleagues. This series of posts is my attempt to move the conversation forward a little.

It seems to me that the first step is to define what we actually mean when we say that digital work should “count” in higher education.

At most colleges and universities around the United States (and to varying degrees elsewhere in the world), there are three domains of activity that faculty members engage in–research, teaching, and service. Most of us have to turn in an annual report that is organized into three sections corresponding to these domains. And in varying ways at various campuses, what can be claimed in each domain is defined by the institution or by departments. Sometimes, those things that count are defined in union contracts. Sometimes they are defined as they come up. In short, there is no standard practice in academia, other than to generally rely on research, teaching, and service as the main categories for faculty evaluation.

Then we get into the thornier issue of how activity in each of these domains is evaluated. Here we see even more variation in practice from one campus to another, from one department to another. What “counts” at one place, is ignored or even penalized at another. At one institution research trumps all, while at another, teaching is the coin of the realm. In some history departments it is enough to have published a book, while in others that book needs to be published by some relatively short list of prestigious presses. Context is everything in this discussion.

Does this mean it is hopeless to even take on the issue of how digital work might fit into such a heterogeneous set of practices?

By no means.

In the history business, we have a very informal and fluid set of standards for determining what is and isn’t meritorious. We all know that an article published in a journal judged to be prestigious is probably more praiseworthy than one published in a backwater journal with little or no reputation. And we know that a book published by a university press that has a great reputation is almost surely better than one published by a press no one has ever heard of.

Or at least we think we know these things.

Whether book or article X published by a prestigious journal/press is actually better than book or article Y published by a journal/press we’ve not heard of is an open question. But we assume in advance that X is probably better than Y.

And not without good reason. Those things submitted for publication to a prestigious press/journal are more likely to go through a more rigorous peer review and editorial process than something published in an underfunded and little known press or journal. And the competition to publish in the prestigious venues is keen–submissions of lesser quality get weeded out.

And thus it has been for generations.

As long as historians produced scholarship that was in a form that fit neatly into this model–books or journal articles published after a peer review process–all was well and the system functioned fairly smoothly. Then digital technology invaded the cozy confines of our discipline and things got a lot more complicated.

I am intentionally using the term “digital work” here rather than “digital scholarship” and my choice of words is in no way accidental. Digital work encompasses everything historians do in the digital realm–scholarship, teaching, and service. “Digital scholarship” is a precisely defined (or should be precisely defined) subset of “digital work.”

Thus, if we are going to argue that digital work should count in higher education, we need to define how this work counts in each of the three main domains of scholarly activity. This was something I had to wrestle with myself not too long ago when I came up for tenure here at George Mason. If you care to read my tenure statement, you can see how I parsed the digital work that I do across the categories. Thus, for instance, I claimed my work on large digital projects such as World History Sources and Women in World History as “applied research on student learning and technology” rather than as the equivalent of the book I published in 2006.

In my formal presentation to my department, I argued that these projects were roughly equivalent to a collection of essays that one might edit. After all, these two projects included dozens of essays (and hundreds of website reviews) by scholars of World History that I and my colleagues had to solicit, edit, and then publish. And taken together, these two projects won the American Historical Association’s James H. Robinson Prize, so they must have had some merit.

That I even had to make the case that they were roughly equivalent to something we already know well is evidence of how difficult the problem is.

Before we can even begin to claim that something called “digital scholarship” should count in the research domain of our professional lives, we would do well to define exactly what constitutes “scholarship.” Here, I think we have an easier task. In almost any discipline one cares to name scholarship has the following characteristics: It is the result of original research; it has an argument of some sort and that argument is situated in a preexisting conversation among scholars; it is public, it is peer reviewed; and it has an audience response.

There are exceptions, of course. A novel, a collection of poetry, a work of art, or a piece of music may all count as scholarship in certain contexts. But by and large, the characteristics I’ve described hold for most forms of scholarship.

This means that for digital scholarship to be scholarship it has to have all of these characteristics. But, as Stan Katz, formerly president of the American Council of Learned Societies said in an interview I did with him for this series, digital scholarship “is a new form of scholarship.” In other words, we need to stop trying to pound the square peg of digital scholarship into the round hole of analog scholarship.

We’ll return to this crucial issue later. But for now, I think it’s easier to define what digital scholarship isn’t than to define what it is–especially because as we’ll see, it is an inherently moving target.

I think we would all agree that a course website or a series of lectures created in one’s favorite slideware program do not constitute scholarship. They may well be very scholarly, but on any campus I can think of, this sort of work falls clearly and unequivocally into the teaching domain.

Where it gets trickier is when we consider digitization projects–whether small in scale, or massive, like the Perseus Project or the Valley of Shadow. Each of these excellent and heavily used projects offers scholars, teachers, students, and the general public unique access to their content. But, as Cathy Davidson at HASTAC told me in an interview for this series, “the database is not the scholarship. The book or the article that results from it is the scholarship.” Or, I would add, the digital scholarship that results from it. In other words, I’m not willing to limit us to the old warhorses of the book or scholarly article.

I also want to emphasize that I have tremendous respect for the scholars and teams of students and staff who created these two projects–both of which I use often in my own teaching. But I also have to say that I don’t think either project can be considered “scholarship” if we use the definition I’ve proposed here.

Why not, you might well ask? The reason is fairly simple in both cases. Neither project offers an argument. Both are amazing resources, but neither advances our understanding of particular historical questions. They make it possible for that understanding to advance in ways that weren’t available before, but as Davidson says, it is what results from a project like these that is the scholarship. Thus, for instance, the article published by Valley of Shadow creators William Thomas and Edward Ayers –” An Overview: The Differences Slavery Made: A Close Analysis of Two American Communities,” American Historical Review, 108/5 (2003)–rises to the level of scholarship in our working definition.

Thus far I have laid out a basic argument about what digital scholarship might be and might not be and how it could count in the tenure and promotion process in higher education. How then might we get to a place where digital scholarship might begin to count?

I have already argued that scholarship has the following characteristics: it is the result of original research; it has an argument of some sort and that argument is situated in a preexisting conversation among scholars; it is public, it is peer reviewed; and it has an audience response.

I think that almost all historians would agree with this definition because it’s the one we use all the time. We’re comfortable with it, it works for us, and given how used we are to it, many historians (including many I know and respect) argue that there is no need to change it. After all, if it ain’t broke, why fix it? [For more on the evolution of peer review in the history business, see the American Historical Association’s 2005 statement on peer review.]

Alas for our current definition of scholarship, the digital world is undermining our certainties.

The big sticking point is the next to the last part of my definition–peer review. For a century or more, “peer review” in our discipline has meant that the historian produces his or her work (book/article) and submits it for publication. Then, after waiting months (or more likely many months) the historian finally receives feedback on his or her work and either has a little more work to do, a lot more work to do, or finds the work rejected entirely.

Why won’t this process survive in the digital world? The answer is pretty simple. It just takes too long and does not work in a medium where gatekeeping makes no sense. By it’s very nature, digital scholarship happens in a dynamic space–one where the work is often “self-published” in the sense that a scholar or a group of scholars creates historical work in the digital environment and then it is made available when it’s done (or close enough to done to show other people). Not after a lengthy process of peer review–but when it’s ready to be seen.

Then, and only then, does the peer review begin. The Internet is an open environment, not the closed environment of the publishing industry that we have lived with for many generations. Anyone can publish anything online and that, of course, means that a lot of dreck appears. But the fact that dreck is scattered all over the Internet does not mean that quality work cannot also be appear through the same process.

The AHA is proposing to try to act as some sort of gatekeeper for digital historical scholarship, but as I’ve already argued, this proposal is doomed because it is trying to find a way to fit the old system into a new technological environment where gatekeeping as we’ve known it doesn’t (and can’t) work.

Already in other industries we have seen what happens when the guardians of the old ways try to hold back the tide of change. Sales of music CDs continue to drop like a stone while sales of individual tracks through services like iTunes continue to rise rapidly. Two decades ago Kodak employed something like ten times as many workers as it does today (when was the last time you bought a roll of film?). And while Amazon.com hasn’t killed off all local bookstores, there certainly are fewer than there used to be.

So what, you might ask? Why do we have to change?

Because if we don’t, we’ll eventually become irrelevant. Already other disciplines that are not as resistant to change have embraced the digital world to a much greater extent. See, for instance, a recent article in the New York Times on the Social Science Research Network. Work posted on this network “counts” in many academic departments around the country despite the fact that peer review takes place after the fact, not before. And in other disciplines–computer science, biology, physics, etc.–peer review takes other forms entirely. So why are we so hung up on keeping a system that made good sense 100 or 50 years ago, but makes less and less sense today?

I wondered what a provost might think about this issue so I interviewed Peter Stearns, our Provost here at George Mason. For those of you who don’t know him, he’s a past vice president of the American Historical Association (Teaching Division), the founding editor of the Journal of Social History, and the author of more than 100 books. So he knows something about peer review.

What he told me was that being a Provost meant that he had to take a much more capacious view of peer review, because each discipline at the University has its own standards for what constitutes proper peer review. What Peter cares about is not how the peer review happens, but that it does happen. “It can be either before or after publication,” he said in our interview.

Other disciplines do it, so what is so particular, so unique about historical scholarship that it must be reviewed prior to publication? I can’t actually think of anything.

I’m not proposing that we throw out a system that has worked for so long in one fell swoop. But I am suggesting that there needs to be a serious discussion in our profession of what peer review means, what its value is to the process of advancing knowledge, and how it can change to take into account the new realities of the digital world. If we don’t have this discussion–and soon–we’re in danger of losing touch with a rising generation of young scholars who will see us as nothing more than cranky old scholars who are hanging onto an old system because it serves our interests and not theirs.

[NB: This series of posts has also been published, in edited form, in Hacking the Academy in the section “Hacking Scholarship.”]