Weak baselines and reporting biases lead to overoptimism in machine learning for fluid-related partial differential equations
<|reference_start|>Weak baselines and reporting biases lead to overoptimism in machine learning for fluid-related partial differential equations: One of the most promising applications of machine learning (ML) in computational physics is to accelerate the solution of partial differential equations (PDEs). The key objective of ML-based PDE solvers is to output a sufficiently accurate solution faster than standard numerical methods, which are used as a baseline comparison. We first perform a systematic review of the ML-for-PDE solving literature. Of articles that use ML to solve a fluid-related PDE and claim to outperform a standard numerical method, we determine that 79% (60/76) compare to a weak baseline. Second, we find evidence that reporting biases, especially outcome reporting bias and publication bias, are widespread. We conclude that ML-for-PDE solving research is overoptimistic: weak baselines lead to overly positive results, while reporting biases lead to underreporting of negative results. To a large extent, these issues appear to be caused by factors similar to those of past reproducibility crises: researcher degrees of freedom and a bias towards positive results. We call for bottom-up cultural changes to minimize biased reporting as well as top-down structural reforms intended to reduce perverse incentives for doing so.<|reference_end|>
@article{mcgreivy2024weak,
    title={Weak baselines and reporting biases lead to overoptimism in machine
  learning for fluid-related partial differential equations},
    author={Nick McGreivy, Ammar Hakim},
    journal={arXiv preprint arXiv:2407.07218},
    year={2024},
    doi={10.1038/s42256-024-00897-5},
    archivePrefix={arXiv},
    eprint={2407.07218},
    primaryClass={math.NA cs.LG cs.NA physics.flu-dyn}
}