Rookie review

What the novice peer reviewer needs to know before combing through a submission.

Claudio Casola had no idea that journal editors had consistently rated his manuscript reviews highly. Then he received an award from Amsterdam-based publisher Elsevier for his “exceptional contribution to the quality of the journal Gene”. Casola is meticulous; he goes through every figure and table of a manuscript, tactfully suggests improvements and always meets his deadlines. He attributes his expertise to practice, publishing experience and immersion in the literature.
Casola, a postdoc in evolutionary genetics at Indiana University in Bloomington, says that his first review, in 2006, was typical of rookie referees. He spent more than 10 hours on the manuscript, poring over the details and asking faculty members for advice. After reviewing more than two dozen papers in the past five years, he has been able to cut the process down to three hours, quickly assessing the originality and merit of a paper. “Reviewing manuscripts makes me feel like I'm a fully fledged member of the scientific community,” says Casola.

BELLE MELLOR

“Young scientists should get involved in the process as they start building their careers, particularly since reviewers are harder and harder to find,” says Bart Wacek, an executive publisher in charge of Elsevier's genetics portfolio based in Cambridge, Massachusetts. Young reviewers are certainly sought after. “The best referees are postdocs,” says Leslie Sage, a senior physical-sciences editor at Nature in Washington DC. “They are at the top of their game, well versed in the literature and politically naive enough to tell the truth.”
Yet Casola, like many graduate and postdoctoral students, was never taught how to review a manuscript; most peer reviewers learn journals' needs and the reviewer's role only through trial and error. Editors' expectations differ according to their fields, but most agree that simply writing thorough, respectful and helpful reviews is the best way for early-career scientists to find their footing and avoid mistakes.

Peer-review blueprint

When reading a paper, good reviewers will not only dig into the fine details of the research, but also assess the article's contribution to the big picture of science. They should keep in mind what editors want most from them: detailed comments that justify their criticisms and are returned in good time.
As online publication increases and new periodicals proliferate, journals are struggling to decrease time to publication. Many, including Gene, track reviewer performance and monitor turnaround time; reviewers who consistently return reviews after the deadline probably won't be called on again. “If you don't have the expertise or time to do a good review, decline right away,” says Mark Spigt, a public-health researcher at Maastricht University in the Netherlands. Last year, Spigt wrote a set of detailed guidelines for reviewers, sharing insight gleaned from his own frustrations early in his career (M. Spigt and I. C. W. Arts J. Clin. Epidemiol. 63, 1385–1390; 2010). “If you commit to the review, block out a set amount of time on the calendar and complete it in that window,” he says.
Editors and seasoned reviewers recommend that referees start the review with a summary of the paper's findings and points of interest, to show that they have read and understood the article. They should also include an impact assessment, detailing whether the paper covers a huge or incremental advance, says Bernd Pulverer, chief editor of the EMBO Journal in Heidelberg, Germany. Spigt lists his major comments, then gives a more detailed, thorough account of the research. He addresses central issues such as whether the research question is relevant to the field, the methods are appropriate and the data support the conclusions (see 'The questions to ask').

Box 1: Helpful hints: The questions to ask

• Is this an innovative approach?
• How do the findings relate to other advances in the field?
• Are the methods appropriate to the scientific question being addressed?
• Does the paper adequately cite all the relevant literature?
• Are the data valid and do they support the conclusions? V.G.

Reduce

J. ROBERTS

Mark Spigt: "If you don't have the expertise or time to do a good review, decline right away".

Reviewers, who tend to be well acquainted with the literature, can best assist editors by comparing the manuscript with other advances in the field to ensure that it is saying something new and that no pertinent citations have been omitted — a task that is often overlooked. “In my personal experience, roughly 20% of authors are hiding relevant work in order to make the paper seem more novel,” says Sage.
Manuscript editors appreciate informed assessments of the paper's findings and data presentation. “I like to see technical details — thoughts on the validity of the work, any concerns about technical issues, or potential improvement to the paper,” says Daniel Kulp, director of the editorial division at the American Physical Society in Ridge, New York, which publishes Physical Review Letters, among other journals. The single most helpful thing a reviewer can do, editors say, is to substantiate critical comments with supporting evidence. “What's really frustrating is when a reviewer simply says 'This is not good enough', yet offers no constructive comments to remedy the problem,” says Pulverer.
Some editors want to see thorough scrutiny of the data, with spot-checks of how uncertainties and outliers are addressed, for example. The level of checking required varies, so novice reviewers should ask editors what they look for. Some editors send cover letters detailing their expectations, but others don't bother.
Sage says it is not uncommon for his reviewers to download raw data to check that they are robust — especially if they are available from a public archive. He doesn't expect that much for every paper, but says that it can be helpful for a reviewer to take the initiative.

Detailed analysis

Some fields and journals encourage extreme thoroughness. “We don't simply accept or reject; we have a detailed editing process that puts [the journal] on the high end of the involvement spectrum,” says David Schimel, editor-in-chief of Ecological Applications and a senior scientist at the National Center for Atmospheric Research in Boulder, Colorado. Reviewers for Ecological Applications often submit a several-page report in which they not only evaluate methodology in considerable detail, check calculations and confirm whether the conclusions are supported, but also make clarifying comments on a line-by-line basis, he says. However, editors at other journals prefer reviewers not to spend their time on that level of detail, fearing that they will burn out.
Astronomy journals are generally comfortable with papers being revised several times, says Chris Sneden, an astronomer at the University of Texas at Austin and editor of The Astrophysical Journal Letters. “It's rare, but a paper can go through five or six review rounds if it starts out as a disaster,” he says. “But the sociology of the field is happy with a lot of back and forth with the author during the process.” This is less common for biology journals, which tend to have many more manuscripts moving through the publication process at any one time.
Perhaps the biggest misconception about reviewing is that referees haven't done their job if they can't come up with enough criticisms. Overzealous reviewers may even allow criticism to devolve into cattiness in their eagerness not to be seen as a pushover. Postdocs are particularly prone to such errors, say some manuscript editors — perhaps because they are subconsciously trying to prove their own worth, or are used to journal-club sessions designed to pick papers apart. A lot of new reviewers simply can't write a review that essentially says 'this is a great paper' — which is perfectly valid, as long as reviewers summarize the key points and express in detail why the techniques and conclusions are sound,” says Kulp. Hemai Parthasarathy, a former manuscript editor at Nature and PLoS Biology and now a senior partner at Torch Communications in San Francisco, California, advises that reviewers “start from a place of respect”, and recognize that the author put a lot of work into the manuscript. “
Another mistake is to ask for lots of extra lab work (see 'What reviewers should know about reviewing'). “There are always more experiments that can be done to nail something down, and reviewers sometimes fall into the trap of asking for them all,” says Parthasarathy. Novice reviewers have to learn to gauge when such requests are justified. “One of the trickiest things for junior people to learn is how meaty a manuscript needs to be,” says Michael Veeman, a molecular biologist who will start his first lab at Kansas State University in Manhattan this month. Parthasarathy says that reviewers should take a step back and think about what is reasonable to expect as proof in a single paper.

Box 2: Inside information: What reviewers should know about reviewing

There are many common misconceptions about the review process. Here are some tips on the reviewer's role, what to do and what to avoid.

• Referees don't accept or reject papers; their comments guide the editor's decision.
• Reviews should not tear papers apart; it is better to offer constructive ideas on how to improve them.
• Reviewers should avoid concentrating on what the study could show in principle. The focus should be on what it actually shows.
• The stature of the scientist being reviewed is unimportant. Referees should be neither hypercritical nor obsequious.
• Judging the science properly means setting aside one's own scientific biases or predilections.
• The reviewer should aim to improve the paper, not make someone else's work their own.
• Reviewers should not waste too much time trying to improve the writing.
• The journal editor will be able to clarify any uncertainties about conflicts of interest or the level of scrutiny expected. V.G.

Expand

Consistent communication

When a revision comes back, reviewers must be consistent. For example, if the authors added an experiment, it would be unreasonable for reviewers to ask for yet more data. “Authors report that they feel that the goalposts are moved during revisions, which gets frustrating,” says Wendy Lipworth, a sociologist who has studied the ethics of scientific review at the University of Sydney in Australia.

A. M. LAWING

Claudio Casola: "Reviewing manuscripts makes me feel like I'm a fully fledged member of the scientific community".

The most frustrating rookie offence, however, might be making contradictory assessments in a single review. Kulp says it drives editors “insane” when a reviewer submits highly critical comments with a recommendation to “publish as is”. Such reviews are most common when journals allow reviewers to submit one set of comments to the editor and another to authors. Contrasting reviews create problems for everybody concerned, says Parthasarathy. At best, they make the editor's decision harder; at worst, the catty ones can start a feud. Reviewers should never write anything that would be damaging if their identity were revealed.
Reviewers should avoid overestimating their own capacity to review multidisciplinary papers: a lack of understanding could lead the reviewer to recommend that perfectly good work is rejected, says Erik De Schutter, a theoretical neurobiologist at the Okinawa Institute of Science and Technology in Japan. He is editor-in-chief for Neuroinformatics, where he has published a paper on his difficulties in getting theoretical modelling papers accepted in general neuroscience journals, many of which insisted on experimental data (E. De Schutter, Neuroinformatics 6, 253–255; 2008). Being qualified to make comments on only one part of a paper doesn't rule out valuable contributions, but reviewers should be open about where their expertise lies. “I am a theorist, so quite often when I'm asked to review experimental papers, I make clear that I'm not qualified to judge the methods,” says De Schutter.
Whatever the content of the review, referees should be completely honest about their affiliations and who helped them to write it. Often, senior scientists invite graduate students or postdocs to write or contribute to the review. Most editors don't mind this, as long as they are notified in advance and all contributors are listed. “Finding the proper mix of expertise among reviewers is a careful calibration on the part of the manuscript editor — one that can get screwed up if the reviewer is not who you think it is,” says Parthasarathy. Graduate students have every right to make sure that their mentors explain how they have helped.
Conflicts of interest may sometimes dictate that researchers decline a review — for example, if they have financial stakes in the paper's content or personal ties with the author. Some journals don't mind, as long as all issues are disclosed. “Biases are not all conflicts of interest; and sometimes editors want the perspective of someone inclined to loathe a particular piece of work,” says Lipworth. If, for example, a potential reviewer is actively writing a paper on the same topic, says Sage, it is probably best to decline the review. But if the reviewer's own paper has already been submitted, and therefore is documented in the scientific record, the reviewer can't be accused of stealing ideas.

Getting noticed

The most straightforward way for researchers to become peer reviewers is for their mentors to introduce them to editors, but there are other routes. The best is to do good science and get published; manuscript editors scour citation databases and conferences, looking for young scientists with expertise that might make them a valued reviewer. Scientific meetings are the most appropriate place for would-be reviewers to introduce themselves to editors. In fact, finding bright young reviewers is one of the main reasons journal editors attend meetings. E-mailing an editor out of the blue is a riskier endeavour and can be seen as overly pushy, says Sage.
Novice reviewers might also find that journals run by scientific societies are often short on reviewers, and so are eager to get them involved. The Ecological Society of America in Washington DC, publisher of Ecological Applications, contacts its student members to find reviewers. Others, such as the American Journal of Pharmaceutical Education, encourage potential reviewers to sign up on their website. “I don't have many reviewers for the pharmacokinetics papers so I've been pushing to get people engaged,” says Gayle Brazeau, the journal's associate editor.
Reviewing may seem like a time sink in the short term, but it can have long-term benefits. For one, watching manuscripts evolve through the editorial process can be a valuable experience, says Veeman. And editors sometimes reward thoughtful, articulate reviews with further opportunities. Schimel says that excellent reviewers might be offered seats on a journal's editorial advisory board. And, notes Wacek, reviewing can be an entry point to a career in scientific publishing.
Spigt's reviews helped him to get a position as an assistant editor with the Journal of Clinical Epidemiology, a leading publication in his field. He hopes one day to become an associate editor. “One of the most important things manuscript reviews can do,” says Spigt, “is help the reviewer build relationships with the journals in which they want to publish.”