&gt; The most common reason we are unable to replicate the remaining 45 papers is that the authors do not provide data and code replication files.<p>&gt; We define a successful replication as when the authors or journal provide data and code files that allow us to qualitatively reproduce the key results of the paper.<p>Well this is underwhelming. I mean, sure, they&#x27;re talking about papers in journals for which sharing data and code is required when asked for, and so they have definitely exposed widespread ignorance of the rules, maybe even a refusal to adhere to them... but the replication that is being talked about is &quot;can we download the data, press a button and get the same results the original authors did?&quot; and not &quot;can we run the experiment again or run the analysis independently and get similar or the same results?&quot;<p>Personally I like the the distinction between replication (a new experiment but with the same setup), reproduction (corroborate using different methods) and re-analysis (download the data, run the code, maybe do some additional analysis). This paper is entirely about re-analysis, not about replication or reproduction. (Cf. <a href="http:&#x2F;&#x2F;sequoia.cs.byu.edu&#x2F;lab&#x2F;files&#x2F;reser2010&#x2F;proceedings&#x2F;Gomez%20-%20Replication%20Reproduction.pdf" rel="nofollow">http:&#x2F;&#x2F;sequoia.cs.byu.edu&#x2F;lab&#x2F;files&#x2F;reser2010&#x2F;proceedings&#x2F;Go...</a>)<p>In one sense, failed re-analysis means research cannot even clear the lowest possible bar: you can&#x27;t even check if the analysis produces the numbers that are mentioned in the research. But in another sense, whether or not researchers manage to release their code or not is only very weakly associated with how good that research is. Research might &quot;fail&quot; re-analysis because no code was provided, but survive both replication and reproduction.<p>The authors compare their work with the Open Science Collaboration which recently pointed out so many unreplicable studies in psychology, but this is not a fair comparison at all. The Open Science Collaboration was a huge endeavor and redid a bunch of experiments from scratch. This is just asking authors &quot;give me your data&quot; and checking a mark &quot;did not replicate&quot; if they didn&#x27;t.