Peer review says “bof”
There are some words that are hard to translate in every language. Giving a sense of the French “bof” is one of these tricky cases. Basically, it means something close to “so-so”, but it is better impersonated by the Comics figure of Achille Talon (there is a section of the English wikipedia article on Achille Talon dealing with his favourite quotes that tries to grasp the essence of “bof” as well). “Bof” is an attitude of indifference, disdain, not wanting to acknowledge reality. And “bof” has to do with the data journals session at the Prague DARIAH HaS Winter School I taught yesterday. Not with the attitude of the participants, at all – boy, were they enthusiastic. But it has to do with our discussion of peer review.
The session was split in a frontal presentation and a hands-on session. In the frontal part, I elaborated on the idea that we need to reconcile the research process and the reviewing process so that reviews actually help improving the research process. The point of my talk was to think about how data journals for the humanities could achieve this goal to some extent, and then try out a data journal sandbox in the episciences platform. But before I got to this, I presented a few general thoughts the readers of this blog already know, namely 1) that one of the key problems in the evaluation system is how it does not consider collaborative endeavours for what they are (namely not an exception, but rather the rule, which means that it should be much more natural to take distributed contributions to a scholarly work into account at all levels) and 2) that pre-publication peer review is not doing its job any more. I don’t have a definitive answer to what we should have instead of pre-publication peer review, I simply stated a few well-known problems: it is not innovative, it contributes to keeping hierarchies in place, it does not contribute to progress or to change as much as we could expect from something that is literally eating up our scholarly time.
I thought that I didn’t need to get any further into a rant against peer review, but the winter school participants were not afraid of doing so. This is how I was told about one peer review deciding on the acceptance of a paper at a conference – only one and no other – that simply consisted in a “bof”, a shrug. As an editor, as a conference reviewer, as a program committee member, this is something that to my eyes totally disqualifies someone as a reviewer. I would not even dare sending such a “review” to an author. But obviously, some did dare. And truly, I can’t blame the young researchers for thinking this could only be a review from an old professor addressed to a young researcher and one more proof that this has all very much to do with perpetuating academic hierarchies. Hearing the other many, many ways in which they suffer from academic disdain was depressing, and I can only hope that the everlasting reproduction of academic hierarchies will stop plaguing them at some point. Obviously, data journals are not the answer to that problem, not in the short or mid-term anyway.
Data journals, though, proved to be a good opportunity to familiarise them with editorial work from the inside. Although the concept of data papers was mostly unknown to them and it was obviously difficult to grasp how the data paper is only a key to understanding a data set and it is the data set that matters in the evaluation process, thinking about how to review data papers allowed to go through a series of editorial challenges: defining a rating grid, a journal rationale, a review system allowing more or less permeability between authors, reviewers and editors. Clever solutions were formulated in order to allow language diversity based on existing structures, a thorough rationale, a thought-through splitting of tasks in the editorial process, a way of finding resources relevant for an episciences-based data journal came out of this (to my eyes extremely fruitful) hands-on session.
While hearing this feedback of the participants on their work setting up a data journal workflow in the hands-on session, one word dominated: standards. This is what we all agree on and proceed with for further work. And it is not “bof”.