Leakage and the reproducibility crisis in ML-based science
this is sort of one of the weird problems that shows up at the intersection between science in the public interest and a market driven system of production.<p>pure science that is publicly funded in the public interest would publish all raw data along with re-runnable processing pipelines that will literally reproduce the figures of interest.<p>but, the funding is often provided by governments with the aim of producing commercializable new technology that can make life better for society.<p>the problem is that if you do the science in the open, then it can be literally picked off by large incumbents before smaller inventors have a chance to try and spin up commercialization of their life&#x27;s work.<p>so we have this system today where science is semi-closed in order to protect the inventors, but sometimes to the detriment of the science itself.