A competitive baseline for deep learning enhanced data assimilation using conditional Gaussian ensemble Kalman filtering
<|reference_start|>A competitive baseline for deep learning enhanced data assimilation using conditional Gaussian ensemble Kalman filtering: Ensemble Kalman Filtering (EnKF) is a popular technique for data assimilation, with far ranging applications. However, the vanilla EnKF framework is not well-defined when perturbations are nonlinear. We study two non-linear extensions of the vanilla EnKF - dubbed the conditional-Gaussian EnKF (CG-EnKF) and the normal score EnKF (NS-EnKF) - which sidestep assumptions of linearity by constructing the Kalman gain matrix with the `conditional Gaussian' update formula in place of the traditional one. We then compare these models against a state-of-the-art deep learning based particle filter called the score filter (SF). This model uses an expensive score diffusion model for estimating densities and also requires a strong assumption on the perturbation operator for validity. In our comparison, we find that CG-EnKF and NS-EnKF dramatically outperform SF for a canonical problem in high-dimensional multiscale data assimilation given by the Lorenz-96 system. Our analysis also demonstrates that the CG-EnKF and NS-EnKF can handle highly non-Gaussian additive noise perturbations, with the latter typically outperforming the former.<|reference_end|>
@article{malik2024a,
    title={A competitive baseline for deep learning enhanced data assimilation
  using conditional Gaussian ensemble Kalman filtering},
    author={Zachariah Malik, Romit Maulik},
    journal={arXiv preprint arXiv:2409.14300},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.14300},
    primaryClass={stat.ML cs.LG math.DS physics.ao-ph}
}