Cargo Cult Science (1974)
In CS (Systems, not Theory) research there&#x27;s a big push now to move toward reproducibility by publishing code and dependencies; <a href="https:&#x2F;&#x2F;twitter.com&#x2F;kragen&#x2F;status&#x2F;727220736633524228" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;kragen&#x2F;status&#x2F;727220736633524228</a> talks a bit about some of the historical context and details.<p>Hackademia has a couple of synergistic efforts going on, directed mostly at securing the free-software distribution infrastructure and somewhat at simplifying system administration: on one hand you have reproducible builds being done by Tor, Mozilla, and Debian, and on the other hand you have Nix, Guix, and the like pushing purely-functional content-addressable package management.<p>The upshot of all of this is that, in theory, in the near future, it will be practical to publish machine-executable instructions for reproducing the bit-identical executables and other output products you used in your research, at a cost of only a few gigabytes, so they can be exactly reproduced at any time in the future, and then modified to do new experiments; and it will be straightforward to make archival copies of this entire set of dependencies so that it won&#x27;t be lost because some random guy died or decided to pull his modules from npm.<p>Mike Hoye famously complained about how far we are from this in a Tweetstorm the other day: <a href="https:&#x2F;&#x2F;twitter.com&#x2F;mhoye&#x2F;status&#x2F;725010388547330048" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;mhoye&#x2F;status&#x2F;725010388547330048</a><p>The Sprite-LFS affair I mention in that tweet is an infamous case of a famous, influential systems paper by a famous, influential academic (Ousterhout) that badly failed reproduction when a different famous, influential academic (Seltzer) tried to reproduce its results by writing a new log-structured filesystem.  Nobody really knows what went wrong, but our best guess at this point is that the Sprite operating system had unknown bugs in it that gave a performance advantage to log-structured filesystems.<p>(ChuckMcM may want to weigh in, since his company at the time (NetApp) was getting record-setting performance with WAFL, which is very similar to a LFS in some ways, but not others.)<p>Jan Vitek, Jean Yang, Sam Tobin-Hochstadt, and Shriram Krishnamurthi are four prominent researchers who are advocating this change; they and others have created a new institution called &quot;Artifact Evaluation Committees&quot;, which review the artifacts submitted by authors to see if they are able to reproduce the results the authors report.  They&#x27;ve written an overview of it here, and they seem to already be getting significant results: <a href="http:&#x2F;&#x2F;jxyzabc.blogspot.com.ar&#x2F;2016&#x2F;05&#x2F;myth-cs-researchers-dont-publish-code.html" rel="nofollow">http:&#x2F;&#x2F;jxyzabc.blogspot.com.ar&#x2F;2016&#x2F;05&#x2F;myth-cs-researchers-d...</a><p>Hopefully over time this will create a bubble of real, reproducible science inside of the systems part of Computer &quot;Science&quot; research.<p>[Earlier versions of this comment did not mention Krishnamurthi or Vitek and the synergistic work in hackademia, which was a combination of cluelessness and carelessness on my part.  I regret the omissions.]