Being cute (portrait of the scientist as a reviewer)
Of course, I could and would work tonight. But my brain is fried after over 6 hours of tele/videoconference today. Also, I had a meeting with the school administration of my district in between telcos and have to think about what it means for the next steps in bringing refugee children to school. So let me tell you some more about my experience as a reviewer for DH-related conferences.
November-december is, traditionally, DH-reviewing time. It all started two years ago, when I volunteered for the DH conference. By then, ConfTool was a weird and complicated interface to me, I had trouble defining my areas of competence and was somewhat worried to have to write 6 reviews, while some were announcing on twitter that they were doing some 20 or so of them. In the first year, I remember my enthusiastic discovery of how advanced English-speaking Digital Humanities project are. In the second year, it did not take me half as much time to write my reviews and I began to be appalled at the gap between my assessment of the papers and that of the other reviewers. What is wrong with me?, I thought. It seemed at first that my judgement was the total opposite of that of my co-reviewers.
Then, and don’t ask me how, things changed. I have reviewed for more conferences, more institutions, the contexts were more various. I had to deal with much more complex review formats than the conference-ConfTool version. I began thinking in terms of review grading giving a paper, a scholar, a chance or not. And then I began to realize how much (and, seen from a different angle, how little) power you have as a reviewer, and what huge importance the definition of your areas of competence and your influence on your choice of papers to review, finally is.
A year ago, I was a program committee member for the first time in the yearly conference of the Digital Humanities in the German-speaking countries. Shortly after that, I had the honor of doing the same for the TEI conference and now again for this year’s conference of German-speaking DH. I am starting to have points of comparison, and also ideas as to what really matters in the reviewing process.
You might remember how, to my big surprise, I was co-opted a member of the board of the DHd at the conference in Graz last February. My main activity in this function so far has consisted in enlarging the reviewer pool so that we could have 3 reviews for each proposal for the next conference. My conclusion of today’s long telephone conference of the program committee (where we could observe the effects of what I have been working towards) is that this is a great help when it comes to putting together a program that is by nature interdisciplinary. Still, the convergence or divergence between three reviews vary a lot. The score difference itself can vary from one reviewer to another, not to mention the wording of the commentaries in the text windows. In the end, we don’t really come around defining what DH has to be, when confronted with reviews saying “this is not DH”. As it turns out, reading reviews and proposals that are tangent to the acceptation rate kind of forces you to define with an extreme precision what a conference or a conference theme is about.
Some of these tangent proposals involved reviews I had written and I could not help notice the differences with my co-reviewers of the same proposals. I tend to give good scores when I find that there is something good in a proposal, and then in the commentary I explain what should be added / improved for the presentation itself. Other reviewers, on the other hand, give harsh scores and in their commentary, hardly formulate a criticism. Today, I came to the conclusion that I am being a cute reviewer. Way too nice, really, when I see the cases that were at debate. I know I have been quite unfriendly in some cases as well, but some of the cases we discussed today were me being way nicer, in the scores I awarded, than the two other reviewers.
This made me take a somewhat critical look at reviewing practice in general. What matters first in the process is the score. Only when the score is relevant (but not overwhelming) do you consider the text to make your final decision. And when it comes to the middle zone between clear acceptance and unavoidable refusal – what we discussed today -, boy, are the texts important!
And this is not just me being cute. With scoring options extremely simplified, the part of the reviews that consist in commentaries is important because it is the place where you can read not only the conception of the discipline carried by the reviewer, but also the values after which he/she assesses the value of the papers. It would really be extremely complicated to formulate this in a pre-defined series of boxes that can be ticked (I am a hands-on kind of Digital Humanist / I am an infrastructure person / I like inspiring projects / I am a computer linguist and this is what DH is about, right? / Please tell me this is about hermeneutics and not computers / etc.). But on the other hand, it is hardly possible for us to get a grip on the latent structure of these texts, which are growing more numerous each year with the increase in the amount of proposals and in the number of reviewers.
I would like to plead for a serious study of these review texts as a key to understand how our community is being built. I know that this is really complicated in terms of right to privacy, etc. But I still think that we would all have something to learn from it, and that it would help us reflect on our reviewing practice in a constructive way. Maybe reviewing would, then, not leave the bitterness many reviewers mentioned to me, but more of a positive feeling of contributing to the community.