[{"document_url": "https://www.nature.com/news/over-half-of-psychology-studies-fail-reproducibility-test-1.18248?error=cookies_not_supported&code=e07c3e3c-a0fa-41b2-997e-b5a898296800", "unformatted_src": "/polopoly_fs/7.29003.1440604264!/image/Resized_nosek6.jpg_gen/derivatives/landscape_300/Resized_nosek6.jpg", "src": "https://www.nature.com/polopoly_fs/7.29003.1440604264!/image/Resized_nosek6.jpg_gen/derivatives/landscape_300/Resized_nosek6.jpg", "formatted_filename": "Resized nosek"}, null, {"document_url": "https://www.nature.com/news/over-half-of-psychology-studies-fail-reproducibility-test-1.18248?error=cookies_not_supported&code=e07c3e3c-a0fa-41b2-997e-b5a898296800", "unformatted_src": "/polopoly_fs/7.25852.1440587393!/image/replicatation-graphic-b.png_gen/derivatives/landscape_300/replicatation-graphic-b.png", "src": "https://www.nature.com/polopoly_fs/7.25852.1440587393!/image/replicatation-graphic-b.png_gen/derivatives/landscape_300/replicatation-graphic-b.png", "formatted_filename": "replicatation graphic b"}, null, null, {"document_url": "https://www.nature.com/news/over-half-of-psychology-studies-fail-reproducibility-test-1.18248?error=cookies_not_supported&code=e07c3e3c-a0fa-41b2-997e-b5a898296800", "unformatted_src": "/polopoly_fs/7.34623.1457002102!/sitegraphics/693197012.png_gen/derivatives/default/693197012.png", "src": "https://www.nature.com/polopoly_fs/7.34623.1457002102!/sitegraphics/693197012.png_gen/derivatives/default/693197012.png", "alt_text": "new-pod-red"}, null]
Brian Nosek's team set out to replicate scores of studies. 

Replication failure

The work is part of the Reproducibility Project, launched in 2011 amid high-profile reports of fraud and faulty statistical analysis that led to an identity crisis in psychology.

But, he adds, “We can really use it to improve the situation rather than just lament the situation. The mere fact that that collaboration happened at such a large scale suggests that scientists are willing to move in the direction of improving.”

The work published in Science is different from previous papers on replication because the team actually replicated such a large swathe of experiments, says Andrew Gelman, a statistician at Columbia University in New York. In the past, some researchers dismissed indications of widespread problems because they involved small replication efforts or were based on statistical simulations.

But they will have a harder time shrugging off the latest study, says Gelman. “This is empirical evidence, not a theoretical argument. The value of this project is that hopefully people will be less confident about their claims.”

Publication bias

The point, says Nosek, is not to critique individual papers but to gauge just how much bias drives publication in psychology. For instance, boring but accurate studies may never get published, or researchers may achieve intriguing results less by documenting true effects than by hitting the statistical jackpot; finding a significant result by sheer luck or trying various analytical methods until something pans out.

Nosek believes that other scientific fields are likely to have much in common with psychology. One analysis found that only 6 of 53 high-profile papers in cancer biology could be reproduced2 and a related reproducibility project in cancer biology is currently under way. The incentives to find results worthy of high-profile publications are very strong in all fields, and can spur people to lose objectivity. “If this occurs on a broad scale, then the published literature may be more beautiful than reality," says Nosek.

The results published today should spark a broader debate about optimal scientific practice and publishing, says Betsy Levy Paluck, a social psychologist at Princeton University in New Jersey. “It says we don't know the balance between innovation and replication.”

The fact that the study was published in a prestigious journal will encourage further scholarship, she says, and shows that now “replication is being promoted as a responsible and interesting line of enquiry”.










