Over the years I’ve both written and received lost of peer reviews for academic journal papers. Writing them is a bit of an art. There is no formal training. Instead people seem to muddle their way through.
What makes for a ‘good’ or ‘bad’ review? Like many, at times I wonder about the merits of the whole concept of anonymous peer review (see for example Les Back’s critique). But it is the system we have and it doesn’t look like going away any time soon. So below are some suggestions for how to write peer reviews. Most of it is advice to myself as much as to anyone else. Comments and additions would be very welcome.
1. Engage with the work on its own terms. The point of peer review is to make a piece of scholarship the best it can be, not to allow you to mould it according to your own interests.
The paper you are reviewing is the paper the author wants to write, not the paper you want to write on the same topic. Don’t confuse the two.
I’ve had reviews that were so specific in their alternative vision for what my paper should become, they crowded out my own sense of the work and what it was trying to do, and were therefore unhelpful.
2. Don’t write reviews when you’re angry. A review is not the place to vent your annoyance at the author for not referencing your own work, using terms you dislike, or anything else. Being passive-aggressive is also best avoided.
If a paper has left you feeling annoyed, wait until you’ve calmed down to write the review.
3. Try to be constructive. Start the review with what you thought worked well. Even if you find the paper to be badly flawed, there should be some strengths in there, otherwise the editors wouldn’t have sent it out for review.
After that, instead of focussing on what is ‘wrong’, make specific constructive suggestions for what should be done differently. So, instead of “this paper shows a disappointing lack of engagement with recent literature”, it’s more helpful to suggest that “the paper would benefit from a much more thorough grounding in recent literature such as…” and then list some references.
(Constructive feedback is basic good practice when marking students’ assignments, so it shouldn’t be too hard to apply that to peer reviewing too.)
Try to avoid diagnosing problems for which you aren’t willing to offer possible solutions. For example, if you think the paper needs more grounding in the literature, make sure you give specific references, with author names and years, so it is clear what literature you are referring to.
Likewise, if you’re suggesting lots of things that need to be added, and the paper is already up to the word limit, try to make some brief suggestions about what could be cut to make space for the additions.
4. Invite rather than insist. Reviews issue forth from an anonymous void. That format is intrinsically alienating. Filling them with exacting demands for “required changes” is probably not the best way to inspire authors to think more deeply, to refine their analysis, to make their work as good as it can be.
Authors have egos, just like everyone else, and are more likely to be receptive to comments framed as suggestions, questions or possible points for consideration, rather than as a list of instructions. One way to do this is to pose queries as multiple alternative possibilities, rather than insisting on a particular direction: “It would be helpful to have a stronger conceptual framework. For example, the authors might draw on [theory x], or [theory y], or some other set of ideas that would help them to theorise [z].”
5. Make it short. This is something I really struggle with. I have a tendency to go to town, listing every possible shortcoming. But as an author, such reviews are tiresome to wade through and can come across as pompous. Your review will probably be one of two or three, and if everyone writes reams, the authors will be drowned in feedback.
Short is merciful. No-one wants to read lengthy treatises on what’s wrong with their work. What most authors and editors are looking for is a concise assessment of key strengths and weaknesses, and some constructive suggestions for improvement.
I’m now aiming to keep my reviews to no more than one A4 page. I’ll try to save more detailed dissections for co-authored work, collaborative projects, PhD students – situations where I have enough involvement, and enough interpersonal contact, to give more in-depth comments a better chance of making a positive difference.
Of course everyone has their own style of reviewing. I recently saw a review that went into incredible depth. In included a massive list of specific things, like corrections for a PhD thesis. It was undeniably impressive, and based on the author’s response it seemed to have been genuinely useful. But who has the time and energy to do this for every paper they review?
6. Ask the author some questions. Not rhetorical questions, or questions to which you already know the ‘right’ answer, but some genuine questions that you would like the author to consider, because you want to hear what their response would be.
Questions can compensate for the limited dialogue in the review process. Aim for questions that directly relate to the subject matter, and which, if the authors could answer them, would extend their analysis and contribute something useful to the field.
7. Be timely. It’s shocking how long the review process can take. I recently waited over 5 months just to have some revisions approved. The journal apologised profusely, explaining that they are at the mercy of reviewers, “and sometimes that mercy is incredibly slow.” It’s very common for editors to apologise for having been let down by reviewers.
So when asked to review, it’s wise to think honestly about whether or not you can take it on given your existing commitments. Some people say that you should take on all the reviews you’re offered, especially if you are an early career researcher. But if reviews are coming in regularly, that is going to add to a lot to your workload, and you might end up not delivering. I recently heard of a Professor who agrees to one review every couple of months, and no more. That seems like a sensible amount.
If you really don’t have time to do it, turn the offer down politely. Or at least explain to the editor that you’d like to do it but it’ll take x weeks, see if they are OK with that. If you take it on, but then realise you can’t meet the deadline, do the decent thing and let the editors know as soon as you can, rather than leaving them to chase you.
Everyone misses deadlines. I’m as guilty of it as anyone. But keeping this to an absolute minimum with reviews would make publishing more enjoyable all round.
8. Have a bit of humility. I have a tendency to take my role as a reviewer too seriously. So these days I’m trying my best to keep in mind that I’m just reviewing one paper, in one journal, which in all likelihood won’t be read by very many people even if it does get published.
Of course this depends on your field. If you’re a medical researcher, your review decision could have consequences for people’s health. But with the journals I review for – in geography, education, childhood studies – the stakes are a lot lower. In most cases, the worst that could happen is that an OK paper gets rejected or a not-very-good paper gets published.
In any case, in the end it’s up to the editors how seriously they take your comments. Plus the authors can always argue against your suggestions in their revisions letter. The role of the reviewer is as an informed advisor to the editors and authors, which is actually quite a modest remit.
9. Be honest. If you think the paper really is nowhere near publishable standard, then be clear about that, explaining why without being rude. Likewise if you think it’s brilliant then say so.