Understanding Stochastic Natural Gradient Variational Inference
<|reference_start|>Understanding Stochastic Natural Gradient Variational Inference: Stochastic natural gradient variational inference (NGVI) is a popular posterior inference method with applications in various probabilistic models. Despite its wide usage, little is known about the non-asymptotic convergence rate in the \emph{stochastic} setting. We aim to lessen this gap and provide a better understanding. For conjugate likelihoods, we prove the first $\mathcal{O}(\frac{1}{T})$ non-asymptotic convergence rate of stochastic NGVI. The complexity is no worse than stochastic gradient descent (\aka black-box variational inference) and the rate likely has better constant dependency that leads to faster convergence in practice. For non-conjugate likelihoods, we show that stochastic NGVI with the canonical parameterization implicitly optimizes a non-convex objective. Thus, a global convergence rate of $\mathcal{O}(\frac{1}{T})$ is unlikely without some significant new understanding of optimizing the ELBO using natural gradients.<|reference_end|>
@article{wu2024understanding,
    title={Understanding Stochastic Natural Gradient Variational Inference},
    author={Kaiwen Wu and Jacob R. Gardner},
    journal={arXiv preprint arXiv:2406.01870},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.01870},
    primaryClass={cs.LG stat.ML}
}