Publication prejudices in the peer review system: A replication study.
Publication bias is one of the main problems in science, because it renders published results less trustworthy. Over 40 years ago, Michael Mahoney conducted a seminal experiment to test the origins of publication bias. He asked 75 journal reviewers to referee manuscripts which described identical experimental procedures, but, depending on experimental condition, reported different results. The results showed that compared to reviewers who evaluated the manuscript with negative results, reviewers who evaluated the manuscript with positive results gave on average higher ratings for methodology, data presentation, scientific contribution, and overall recommendation.
However, there are some reasons that warrant a thorough replication of this highly cited study. First, it is not known whether these results can be generalized to other psychology journals and to the current time. Secondly, the sample size was quite small. Given the increasing interest in biases in science, and the central role of reviewers in them, it is crucial to replicate this study now in various journals, using a large and rigorously set up study.
We therefore seek to replicate this study with a new and larger sample, using a rigorous pre-registered design. In the first phase we will study how to best update the study materials and investigate the influence of these changes in a series of pilot studies. Subsequently, we will investigate the main question to be replicated: Do reviewers evaluate manuscripts with positive results differently than manuscripts with negative results? Furthermore, we extend this study to investigate whether reviewers prefer significant results (e.g., positive-outcome bias) or results that are in line with their own beliefs (e.g., confirmation bias). Moreover, this study will shed light on the possible effectiveness and relevance of registered reports by investigating whether reviewing only the introduction and method section will reduce bias against negative results.