  Progress in hybrid physics-machine learning (ML) climate simulations has been
limited by the difficulty of obtaining performant coupled (i.e. online)
simulations. While evaluating hundreds of ML parameterizations of subgrid
closures (here of convection and radiation) offline is straightforward, online
evaluation at the same scale is technically challenging. Our software
automation achieves an order-of-magnitude larger sampling of online modeling
errors than has previously been examined. Using this, we evaluate the hybrid
climate model performance and define strategies to improve it. We show that
model online performance improves when incorporating memory, a relative
humidity input feature transformation, and additional input variables. We also
reveal substantial variation in online error and inconsistencies between
offline vs. online error statistics. The implication is that hundreds of
candidate ML models should be evaluated online to detect the effects of
parameterization design choices. This is considerably more sampling than tends
to be reported in the current literature.

Atmospheric and Oceanic Physics;Machine Learning