Prestigious Science Journals Struggle to Reach Even Average Reliability
The issue the article doesn&#x27;t spend too much time on one of the major reasons why reliability is so low: there are too many confounding environmental variables to control for, so work done in one lab often doesn&#x27;t generalize<p>One seemingly standard, innocuous piece of equipment can make immune cells &quot;aggressive&quot;, while a different brand can make them tolerant. No one knows why. Genetically identical mice ordered from one vendor can consistently have different disease progression compared to mice from another vendor. Background noise in animal facilities can mess with experimental outcomes and even cure disease. Two lots of otherwise identical reagents can act almost like different chemicals<p>Many findings are the result of some set of confounding variables rather than the independent variable. It is impossible to know these things in advance or control for them. The best tactic is to do the same experiment in a different lab with a different research team to control for all these &quot;other factors&quot;. Which is expensive (often six of seven figures) and can take months or even years. It is best practice when startups license tech from a university to ask for an exclusivity period where they reproduce the key experiments before committing to a full license<p>Imagine that for every open source library you use, all you have is a detailed read me, a set of tests, and some infrequently commented white board code. You reconstruct the code as best you can (which isn&#x27;t that hard), but for some reason the tests don&#x27;t check out. The bug isn&#x27;t with your software. You call the original researchers and find they used a few libraries that you didn&#x27;t, and these didn&#x27;t show up in the publication. You have to manually install all these libraries, test them, and integrate them. Still doesn&#x27;t work. You realize they used a different os and have to account for that. After six months, still nothing. After visiting their office, you realize the computer used to write the software is on the floor on some very fuzzy carpet, and there is some weird electrostatic bug in the hardware that contributed to how their software performed<p>This is perhaps not the best analogy, and I&#x27;m not a scientist myself though have managed scientific teams for years, managed a few tech transfers, and evaluated tons of science as potential investments, but I just wanted to drive home the point of how tough science can be. If others can critique this I would welcome it as it would be great for wet lab scientists and engineers to understand each other&#x27;s worlds better<p>Of course there is also fraud, cherry picking of data, and careful manipulation of study design to make externalities work in your favor, but a lot of the reliability &#x2F; reproducibility crisis comes down to lots of uncontrollable &#x2F; unknowable variables and expensive and long development cycles