Today marks the launch of a new initiative in which the Psychology and Cognitive Neuroscience section of Royal Society Open Science guarantees to publish any close replication of any article published in our journal, and from most other major journals too.


Credit: Anastasiya Tarasenko

Replication – it’s the quiet achiever of science, making sure previous findings stand the test of time. If the scientific process were a steam ship, innovation would be sipping cognac in the captain’s chair while replication is down in the furnaces shoveling coal and maintaining the turbines. Innovation gets all the glory but without replication the ship is going nowhere.

In the social and life sciences, especially, replication is terminally neglected. A retrospective analysis of over a hundred years of published articles in psychology estimated that just 1 in 1000 reported a close replication by independent researchers.

What explains this unhappy state of affairs? In a word, incentives. Researchers have little reason to conduct replication studies within fields that place a premium on novelty and innovation. Also, because psychology and neuroscience are endemically underpowered, a high quality replication study usually requires a substantially larger sample size than the original work, calling on a much greater resource investment from the researchers.

Even if researchers summon the necessary motivation and resources to conduct a replication, getting their completed studies published can be a thankless and frustrating task. Many journal editors – especially those who prize novelty – will desk-reject replication studies without them ever seeing an expert reviewer. And those replications that go further face a grueling journey through traditional peer review. Where the replication study fails to reproduce the original findings, proponents of the original work will point to some difference in methods, however trivial, as the cause for the non-replication. And where the replication succeeds, the reviewers and editor are likely to conclude that “we knew this already, so what does the study add”? All paths veer toward the same destination: rejection and the file drawer.

At Royal Society Open Science we believe that overcoming problems with reproducibility in science requires the creation of a new path that recognises and truly values the importance of replication.

Why Registered Reports are not (quite) the answer

Those familiar with the range of reproducibility initiatives currently available in science will be wondering if Registered Reports, which are already offered at Royal Society Open Science, solve the problem of championing replications. While they move us in the right direction, they don’t go far enough.

With Registered Reports, the protocol underlying a study is reviewed before starting data collection or analysis, and the paper is accepted in advance regardless of the eventual results.

This format presents two main weaknesses. First, a replication study submitted through the Registered Reports track can be rejected before results are known, if the reviewers believe the original study contained a significant flaw or oversight, or if the question itself is judged uninteresting or unimportant. This can incentivize authors to change the proposed design, moving it away being a close replication. This invalidates the whole enterprise.

Secondly, Registered Reports are suitable only for proposals where the results are not yet known by the authors. This means Registered Reports can’t unlock the file drawer of existing replications that are already completed and analysed.

Introducing accountable Replications

We agree with Sanjay Srivastava, creator of the Pottery Barn rule of scientific publishing: “you publish it, you buy it”. In his words: “Once a journal has published a study, it becomes responsible for publishing direct replications of that study. Publication is subject to editorial review of technical merit but is not dependent on outcome.” When a journal publishes an empirical study it becomes accountable for the replicability of that study.

How will this policy work in practice? Readers can find our detailed editorial policy for Replications here but the main highlights are:

  • Royal Society Open Science guarantees to publish any close replication of any study previously published in its Psychology and Cognitive Neuroscience section. This commitment extends to replication studies themselves, with no limit on the number of acceptable repeats.
  • For already-completed studies, authors can submit via the Results-Blind track in which the introduction and methods are initially reviewed with the results withheld, and with the editorial decision based on this initial stage of review regardless of the results. For prospective proposals, authors instead submit the introduction and methods via the Fully Preregistered track (our existing Registered Reports workflow with slightly different review criteria).
  • One concern with results-blind review (where results are known to the authors but not the reviewers) is that reviewers may assume that the results are negative or confusing, leading to biased reasoning when assessing the paper. Therefore, reviewers will initially be blinded to whether the article has been submitted via the Results-Blind or Fully Preregistered track. Submissions in both categories will be written in past tense.
  • To achieve acceptance, the study must be sufficiently close to the original work to be considered by expert reviewers to be a replication. It must also have a sufficient sample size, and it must be ethically approved.
  • We are assuming accountability for a range of other major journals that publish research in psychology and cognitive neuroscience (see our detailed policy for the full list). We will also consider, on a case-by-case basis, presubmission enquiries for replications of studies published in a wider range of journals.
  • If there is a flaw in the original study, then depending on the severity of the flaw, an editorial commentary may be appended to the final article noting the methodological limitation. Crucially – and uniquely to this initiative – unless the study being replicated contains a severe flaw AND arises from a paper published somewhere other than Royal Society Open Science, the existence of this flaw will not influence the publication decision. And even then, the flaw will not lead to outright rejection but the requirement to conduct an additional corrected follow-up study by the replicating authors.

Where will this policy take us?

In the short term we hope to see it unlock the vast file drawer of unpublished replications in psychology and cognitive neuroscience.

In the long term, our vision is broader. As Sanjay Srivastava pointed out over five years ago, we must realign the incentives in publishing to make journals accountable for the reproducibility of the work they disseminate. If a journal achieves fame for publishing a large quantity of impactful, novel studies with positive results, but is then bound to publishing attempted replications of those studies, then the journal has a reputational incentive to ensure that the original work is as robust as possible.

We hope to see this model normalise replication studies and give the scientists who conduct them the prominence they deserve. Replication has spent long enough shoveling coal in obscurity. It’s time for a taste of the captain’s chair.


Chris Chambers is a professor of cognitive neuroscience at Cardiff University and subject editor for Registered Reports at Royal Society Open Science.

 

2 Responses to “Reproducibility meets accountability: introducing the replications initiative at Royal Society Open Science”

  1. GBSOD

    While an admirable sentiment it feels a bit hollow. Does this vision extend to other journals published by the Royal Society? It seems that those of us that attempt to replicate the ‘Innovative’ studies published in those journals are limited to Open Science and therefore forced to pay upfront for the privilege of digging at the coalface all the while receiving lower scores for prestige and or impact? I fail to see any real incentive for the individual researchers.

  2. Andrew Dunn

    At present, the Replication article type (http://rsos.royalsocietypublishing.org/page/replication-studies) is limited to Royal Society Open Science. The journal operates a number of open science features, and the Replication articles tie together with Registered Reports (http://rsos.royalsocietypublishing.org/registered-reports) as part of the journal’s Accountable Replication Policy. The journal was launched not only to encourage open science initiatives but also to allow the Royal Society to experiment with other publishing innovations. If this latest approach to tackling some of the challenges faced by scholarly scientific publishing is successful, as we expect it to be, the Editorial Boards of the other Royal Society journals will have the opportunity to discuss and implement the article type if they agree to do so.

    Regarding article processing charges, no author in Royal Society Open Science is required to pay upfront for submission or publication. Only when their paper is accepted for publication will authors be asked to pay an article processing charge (http://rsos.royalsocietypublishing.org/page/charges). Authors able to demonstrate that they are unable to pay the charge in the event their paper is accepted for publication can request that their fee be waived under our generous waiver policy (http://rsos.royalsocietypublishing.org/article-processing-charge-waivers).

    Royal Society Open Science offers a range of journal- and article-level metrics (http://rsos.royalsocietypublishing.org/citation-metrics), with each manuscript having its own metrics tab providing details of how often a paper has been shared, read, and downloaded. A recent example is provided here (http://rsos.royalsocietypublishing.org/content/5/9/181295.article-info)

    If you have additional queries, please direct them to openscience@royalsociety.org, and we’ll be glad to assist.