Why we can ignore reviews of scientific code by commercial software developers
I read the critique in depth. Let me summarize the claims:<p>1. The developers had a couple of bugs that cause non-deterministic outputs in weird settings. It&#x27;s a randomized model so this isn&#x27;t a disaster, but obviously it would be better to have all non-determinism derived from a single seed. However, none of the bugs point to scientific issues. Still, these are real bugs that should be fixed. And indeed: another team pointed out some issues, they triaged the report, and it appears that they&#x27;ve been fixing these issues.<p>2. The code doesn&#x27;t run well multi-threaded, and so the scientists run it on a single core. This seems like a feature the developers have de-prioritized. The critics make a HUGE deal about how using a single core is inefficient, but at the end of the day: if scientists can get simulations done in a reasonable time using a single core, this seems like a valid prioritization of limited development effort. The critics know better, obviously.<p>3. There aren&#x27;t a bunch of unit tests. But as the authors point out code is in development and being altered continuously to reflect new developments in society and new knowledge about the virus, so it seems hard to develop unit tests that won&#x27;t become broken as the underlying components change.<p>4. The critics seem to have a strong grounding in epidemiology. I&#x27;m just kidding, they claim to be pure software engineers, but still insist on spending a significant fraction of their critique disputing the calculation of R0 and lecturing the epidemiologists on how they think it should be calculated. This is a giant red flag given that the critics&#x27; sole stated expertise is software engineering.<p>5. The developers are continuing to add features to simulate new aspects of the ongoing pandemic (e.g., new code to simulate the impact of contact tracing apps) instead of fixing every minor issue in the code. Given that we&#x27;re in the midst of a deadly epidemic that is putting human beings into the ground, this seems like a completely unavoidable resource allocation decision to me.<p>6. The critics can&#x27;t even be bothered to pretend their criticism is objective. E.g., the piece concludes with &quot;on a personal level, I’d go further and suggest that all academic epidemiology be defunded.&quot; Yeah, sure, you&#x27;re a totally disinterested software expert here.<p>I guarantee you if you look at any piece of software -- commercial or scientific, no matter the budget -- with the politically-motivated goal of tearing apart the author, you will succeed. I can say this having spent years doing security and expert witness code reviews for some of the biggest and smallest companies in the software industry. This piece isn&#x27;t a critique, it&#x27;s just a hatchet job. And not even a particularly creative one.