Tag Archives: Peer Review

Making Digital Scholarship Count (2)

In my previous post in this thread, you may have noticed that I used the term “digital work” rather than “digital scholarship.” My choice of words was in no way accidental. Digital work encompasses everything historians do in the digital realm–scholarship, teaching, and service. “Digital scholarship” is a precisely defined (or should be precisely defined) subset of “digital work.”

Thus, if we are going to argue that digital work should count in higher education, we need to define how this work counts in each of the three main domains of scholarly activity. This was something I had to wrestle with myself not too long ago when I came up for tenure here at George Mason. If you care to read my tenure statement, you can see how I parsed the digital work that I do across the categories. Thus, for instance, I claimed my work on large digital projects such as World History Sources and Women in World History as “applied research on student learning and technology” rather than as the equivalent of the book I published in 2006.

In my formal presentation to my department, I argued that these projects were roughly equivalent to a collection of essays that one might edit. After all, these two projects included dozens of essays (and hundreds of website reviews) by scholars of World History that I and my colleagues had to solicit, edit, and then publish. And taken together, these two projects won the American Historical Association’s James H. Robinson Prize, so they must have had some merit.

That I even had to make the case that they were roughly equivalent to something we already know well is evidence of how difficult the problem is.

Before we can even begin to claim that something called “digital scholarship” should count in the research domain of our professional lives, we would do well to define exactly what constitutes “scholarship.” Here, I think we have an easier task. In almost any discipline one cares to name scholarship has the following characteristics: It is the result of original research; it has an argument of some sort and that argument is situated in a preexisting conversation among scholars; it is public, it is peer reviewed; and it has an audience response.

There are exceptions, of course. A novel, a collection of poetry, a work of art, or a piece of music may all count as scholarship in certain contexts. But by and large, the characteristics I’ve described hold for most forms of scholarship.

This means that for digital scholarship to be scholarship it has to have all of these characteristics. But, as Stan Katz said in an interview I did with him for this series, digital scholarship “is a new form of scholarship.” In other words, we need to stop trying to pound the square peg of digital scholarship into the round hole of analog scholarship.

We’ll return to this crucial issue later. But for now, I think it’s easier to define what digital scholarship isn’t than to define what it is–especially because as we’ll see, it is an inherently moving target.

I think we would all agree that a course website or a series of lectures created in one’s favorite slideware program do not constitute scholarship. They may well be very scholarly, but on any campus I can think of, this sort of work falls clearly and unequivocally into the teaching domain.

Where it gets trickier is when we consider digitization projects–whether small in scale, or massive, like the Perseus Project or the Valley of Shadow. Each of these excellent and heavily used projects offers scholars, teachers, students, and the general public unique access to their content. But, as Cathy Davidson at HASTAC told me in an interview for this series, “the database is not the scholarship. The book or the article that results from it is the scholarship.” Or, I would add, the digital scholarship that results from it. In other words, I’m not willing to limit us to the old warhorses of the book or scholarly article.

I also want to emphasize that I have tremendous respect for the scholars and teams of students and staff who created these two projects–both of which I use often in my own teaching. But I also have to say that I don’t think either project can be considered “scholarship” if we use the definition I’ve proposed here.

Why not, you might well ask? The reason is fairly simple in both cases. Neither project offers an argument. Both are amazing resources, but neither advances our understanding of particular historical questions. They make it possible for that understanding to advance in ways that weren’t available before, but as Davidson says, it is what results from a project like these that is the scholarship. Thus, for instance, the article published by Valley of Shadow creators William Thomas and Edward Ayers —An Overview: The Differences Slavery Made: A Close Analysis of Two American Communities,American Historical Review, 108/5 (2003)–rises to the level of scholarship in our working definition.

In my next post, I’ll try to get to the heart of the matter–what would digital scholarship actually look like?

Making Digital Scholarship Count

Today I am inaugurating an extended series of posts on the question of how digital scholarship should “count” in the ways that things count at colleges and universities. As more and more scholars do work in the digital environment they are expecting this work to count toward tenure, promotion, and other types of formal evaluation (in the hiring process, for instance). It is because I have attended numerous conference sessions and meetings over the past several years where this topic has come up (often in very passionate ways), that I have decided to write this series.

The other motivation for the series was an email I received from a friend and colleague whom I respect very much taking issue with something I wrote after my experiences at our THATCamp unconference here at CHNM. In that post I said that one reason why historians doing digital work find it difficult to receive full credit for their digital work (as opposed to other things they do like publishing a book or an article, teaching a class, or providing service to their institution) is that historians are a fussy and conservative tribe generally resistant to innovation.

My friend’s email suggested that this sort of name calling (his formulation, not mine) would only set back efforts to convince historians to accept digital work as “scholarship” on a par with other forms of scholarly work. I’m standing by my characterization of my tribe as fussy and conservative when it comes to innovations–after all, it’s worth remembering how hard it was for women’s history or cultural history to be accepted as a valid approaches to historical scholarship.

But, having said that, I do think that something needs to be done–something other than complaining about my colleagues. This series of posts is my attempt to move the conversation forward a little.

It seems to me that the first step is to define what we actually mean when we say that digital work should “count” in higher education.

At most colleges and universities around the United States (and to varying degrees elsewhere in the world), there are three domains of activity that faculty members engage in–research, teaching, and service. Most of us have to turn in an annual report that is organized into three sections corresponding to these domains. And in varying ways at various campuses, what can be claimed in each domain is defined by the institution or by departments. Sometimes, those things that count are defined in union contracts. Sometimes they are defined as they come up. In short, there is no standard practice in academia, other than to generally rely on research, teaching, and service as the main categories for faculty evaluation.

Then we get into the thornier issue of how activity in each of these domains is evaluated. Here we see even more variation in practice from one campus to another, from one department to another. What “counts” at one place, is ignored or even penalized at another. At one institution research trumps all, while at another, teaching is the coin of the realm. In some history departments it is enough to have published a book, while in others that book needs to be published by some relatively short list of prestigious presses. Context is everything in this discussion.

Does this mean it is hopeless to even take on the issue of how digital work might fit into such a heterogeneous set of practices?

By no means.

In the history business, we have a very informal and fluid set of standards for determining what is and isn’t meritorious. We all know that an article published in a journal judged to be prestigious is probably more praiseworthy than one published in a backwater journal with little or no reputation. And we know that a book published by a university press that has a great reputation is almost surely better than one published by a press no one has ever heard of.

Or at least we think we know these things.

Whether book or article X published by a prestigious journal/press is actually better than book or article Y published by a journal/press we’ve not heard of is an open question. But we assume in advance that X is probably better than Y.

And not without good reason. Those things submitted for publication to a prestigious press/journal are more likely to go through a more rigorous peer review and editorial process than something published in an underfunded and little known press or journal. And the competition to publish in the prestigious venues is keen–submissions of lesser quality get weeded out.

And thus it has been for generations.

As long as historians produced scholarship that was in a form that fit neatly into this model–books or journal articles published after a peer review process–all was well and the system functioned fairly smoothly. Then digital technology invaded the cozy confines of our discipline and things got a lot more complicated.

In the next post in this series, I’ll consider the ways that digital media have complicated and will continue to complicate this system that has served us reasonably well for so long.

Historical Code Monkeys

For years historians engaged in digital work have used the “I can’t really write much code” excuse as a way to avoid everything from learning simple tools like CSS to more complex ones like MySQL and PHP. Alas for me and everyone else who has used the “me no code” excuse, we’ll have to find a new reason why we can’t program.

Bill Turkel and Alan MacEachern have just released The Programming Historian and with its release my excuses go poof. As soon as you go to the website that is the “book” you’ll see that it is a wiki and so an evolving project rather than a static monograph. This approach seems particularly right to me given how fast the technologies we use change. With each change the authors will be able to update their work to keep us all closer to the cutting edge.

You’ll also see that this project is by no means complete. The “coming attractions” list is extensive and over time I expect it will get even longer.

Also of particular note is the fact that the authors provide a list of their peer reviewers. Peer review is an essential part of scholarship, but as we all know, it is mostly blind or double-blind. As befits an open source project, the list of peer reviewers for The Programming Historian is open and available.

If, like me, you have been putting off learning more sophisticated programming skills, this project is the place to start, because it is accessible to the average historian and is written by historians for historians. You won’t get to claim the status of “code monkey” after reaching the end of the current iteration of the project, but keep at it. Someday you too can be a historical code monkey.