Traditional peer review is a relatively private affair, with the referee reports being seen by only a handful of individuals and the name of the referee being seen by even fewer. The move towards greater transparency in science has led to a number of journals adopting a more open model.

Opening up the review process

RSOS_2-9_ThumbnailsOn Royal Society Open Science we operate the following model. At all points in the peer review process we encourage referees to sign their reports, thereby disclosing their name to the author. Where authors agree, we make the editorial process transparent by publishing the referee reports, the decision letter and associated author responses, alongside the actual manuscript. This allows readers to better assess the published paper and provide post-publication comments, which are also published. If the referees have opted to disclose their names, then these are included alongside the published reports. An outline of the possible scenarios is listed below:

Author agrees to open peer review – referee agrees to open peer-review
Signed referee report published

Author does not agree to open peer review – referee agrees to open peer-review
Referee name only disclosed to author, referee report is not published

Author agrees to open peer review – referee does not agree to open peer-review
Referee name not disclosed to author, anonymized referee report published

Author does not agree to open peer review – referee does not agree to open peer-review
Referee name not disclosed to author, referee report is not published

Here’s an example of the first scenario, a paper from August this year on improving body mass estimates in mammals.

188601-FINAL-WEB-READY-1_220x119With Peer Review Week upon us, we decided to ask the editorial board of Royal Society Open Science how they feel about open peer review. Here’s what they said:

  • Transparency is good
  • This is a great idea, it is likely to make peer review more thoughtful and balanced
  • Neutral
  • Mixed feelings. On the one hand, I like the transparency and it tends to discourage biased or unfair reviews. On the other hand, it may put some referees off and make them reluctant to be as honest in their criticisms as they might otherwise be
  • I do not like open peer reviews – there are a lot of vicious people out there

A mixed bag.

4S8A7083Earlier this year, nine months after the launch of Royal Society Open Science, we held a series of lively debates here at the Royal Society on the Future of Scholarly Scientific Communication, including sessions on peer review. You can read arguments for and against peer review here, but we also discussed its future development, evolution and alternative practices.

The following is an edited extract from the full FSSC report which can be accessed here.

Can we reform peer review?

A lot of discussion cantered on the need to separate dissemination from verification, and that peer review should be done directly by the scientific community rather than through publishers who impose high costs on the process. One way of achieving this is by the use of pre-print repositories. Physicists pioneered this with ArXiv in the 90s and findings are often, though not invariably, published in a journal thereafter – we now welcome submissions directly from ArXiv. The advantages of pre-print servers are:

  • Speed; articles are made public immediately
  • Open and collaborative feedback
  • Opportunity to improve a paper before submission to a journal
  • Greater chance of errors being picked up and corrected due to greater exposure pre-publication

A system of universal, free-to-view and well-regulated preprint repositories with good trackback mechanisms for comments was seen by many to be the most desirable model. Of course, there are improvements to be made to a pre-print system too such as ways to provide recognition for reviewers, improving cross search functionalities and ensuring compatibility with existing servers.

The principle of peer review is necessary and valuable

Given how well such systems work in certain communities already – such as physics – the need for journals at all was questioned. Nevertheless, suggestions for reforming peer review based on the existing publisher-journal model were helpfully debated and included:

  • Open peer review as standard with signing of reports optional
  • Use of objective peer review, i.e. reviewing for correctness rather than impact
  • Collaboration between all referees to produce a single review
  • Portable peer review with authors owning their reports and able to take them from journal-to-journal, following a rejection for example
  • Universally recognized systems for crediting peer reviewers

The overall view was that the principle of peer review was necessary and valuable, but should be organized in a different way. In general, it was felt that opportunities offered by new technologies and the web had not yet been fully exploited, and there was a role for learned societies and funders to encourage innovation and drive the necessary changes. We believe that the option of open peer review on journals such as Royal Society Open Science is a step in the right direction.

2 Responses to “How do you feel about open peer review?”

  1. Macrocompassion

    I’m all for open peer review–transparency is one of the ideals associated with the spread of scientific knowledge and once the openness is acknowledged there should be no going back! I wish somebody in the RS would peer review my work on macroeconomics theory, not only because it is a dramatic break-through in what is at last a proper science, but because this knowledge is of great potential use to the treasury parts of government, which (mostly) seek to increase the prosperity through social justice! (Incidentally, the book is titled: “Consequential Macroeconomics–Rationalizing About How Our Social System Works” by David Harold Chester.)

  2. Oooh Lalala

    Not a bad idea, how about author anonymity at the peer-review stage?