Scientists and the design of experiments under scrutiny after a major project fails to reproduce results of high profile studies

Some of the most high profile findings in social sciences of the past decade do not stand up to replication, a major investigation has found.

The project, which aimed to repeat 21 experiments that had been published in Science or Nature – science’s two preeminent journals – found that only 13 of the original findings could be reproduced.

The research, which follows similar efforts in psychology and biomedical science, raises fresh concerns over the reliability of the scientific literature. However, the project’s leaders say their results do not reflect a “crisis” in the social sciences.

Prof Brian Nosek, executive director of the Center for Open Science and a professor at the University of Virginia, said: “I don’t think it’s a crisis, it’s a reformation. We’re in the midst of a dramatic increase in the rigour and transparency of research in the social sciences.”

Nosek said that, while some previous replication efforts had been viewed with hostility, this effort had been collaborative. Authors from nearly all of the papers under scrutiny engaged with the project to help ensure the repeat experiments were close replicas of the originals – and several of these scientists published responses that ran alongside the latest paper.

“It’s not recriminations about who is a bad researcher or otherwise,” said Nosek. “That’s something that has completely changed over the last five years. Replication in 2012 felt like an attack because it was so rare in science at the time. Now it’s become normal.”

Findings that failed to replicate included a study suggesting that viewing a picture of Rodin’s sculpture The Thinker led to people reporting weaker religious beliefs (a possible explanation being that analytical thought, as represented by the sculpture, counteracted religious beliefs). The finding that the physical act of washing your hands leads to less muddled thinking (a phenomenon known as cognitive dissonance) also failed the replication test.

“That doesn’t mean it’s unreplicable, no study is definitive,” said Nosek. “Science is really a process of uncertainty reduction.”

In total, the team tried to replicate one main finding from each of the 21 social science papers published between 2010 and 2015 in Science or Nature, widely regarded as the two most prestigious scientific journals.

They found evidence to back the original conclusions in 13 of the 21 (62%) studies. But, on average, the sizes of the effects recorded were about 75% as big in the replication studies, despite these using sample sizes that were on average five times as big.

“These results show that ‘statistically significant’ scientific findings need to be interpreted very cautiously until they have been replicated even if published in the most prestigious journals,” said Magnus Johannesson of the Stockholm School of Economics, another of the project leaders.

The latest work revealed scientists were also uncannily accurate at predicting which studies would later succeed or fail to replicate. About 200 scientists were recruited and on average predicted the replication outcomes for 18 out of the 21 papers under scrutiny.

Prof Malcolm Macleod, a neurologist at the University of Edinburgh who has previously investigated reproducibility in biomedical science, said there was a need to prioritise the quality of science as well as the novelty of findings. “We need to wean ourselves off the nectar and the crack cocaine of highly exciting results and work out what we can do to maximise the quality” he said. “That’s becoming much more of a thing now.”

Nosek agreed, saying that the growing trend for pre-registering the aims of a study and the increasing number of journals to have adopted policies promoting transparency were encouraging.

“With these reforms, we should be able to increase the speed of finding cures, solutions, and new knowledge,” he said. “Of course, like everything else in science, we have to test whether the reforms actually deliver on that promise. If they don’t, then science will try something else to keep improving.”

The findings are published in the journal Nature Human Behaviour.