Operator Variational Inference
Variational inference is an umbrella term for algorithms which cast Bayesian inference as optimization. Classically, variational inference uses the Kullback-Leibler divergence to define the optimization. Though this divergence has been widely used, the resultant posterior approximation can suffer from undesirable statistical properties. To address this, we reexamine variational inference from its roots as an optimization problem. We use operators, or functions of functions, to design variational objectives. As one example, we design a variational objective with a Langevin-Stein operator. We develop a black box algorithm, operator variational inference ( ), for optimizing any operator objective. Importantly, operators enable us to make explicit the statistical and computational tradeoffs for variational inference. We can characterize different properties of variational objectives, such as objectives that admit data subsampling-allowing inference to scale to massive data-as well as objectives that admit variational programs-a rich class of posterior approximations that does not require a tractable density. We illustrate the benefits of on a mixture model and a generative model of images.
The paper introduces Operator Variational Inference, that defines novel variational objectives by using operators and extends standard variational approaches based on the minimization of the KL divergence. Operator Variational Inference is efficient even without the standard mean field assumption, and it allows to approximate the posterior with a broad class of flexible distributions. When using the reparametrization gradient, the introduced variational objectives do not require the variational distribution q to be analytically tractable, and one can only learn how to sample from it with a Variational Program.
There has been a lot of interest recently in defining variational objectives that improve upon the standard variational bound, as minimizing KL[q||p] leads to variational distributions that severely underestimate the variance of the posterior. By using operators, this paper introduces a novel way to form variational objectives that allow to fit a more flexible class of variational approximations, either by optimizing the parameters of q or by learning how to sample from it (variational programs). I find this approach very interesting, and a promising research direction.  The presentation of the paper, especially in the theoretical part, is sometimes confusing and difficult to understand: - I found section G/H/I/J of the supplementary material necessary to understand more intuitively Operator Variational Inference. They should be put in the main paper, or at least referenced more explicitly and using section names. - In lines 144-161 you only introduce in detail the type of distributions obtained with the reparametrization, but in section 4 you also present the score function gradient. This is confusing, especially as it is only said in the supplementary material that in this case it is needed an analytically tractable variational distribution, as needed in equation 8.  The mixture of Gaussian experiment is useful to understand the behaviour of the different variational objectives, but it looks strange that when using the KL objective the variational approximation does not pick only one of the modes. The MNIST experiment is not enough to assess how the different methods perform. You may consider using more that 10 units (e.g. 100) and reporting how much time it takes for the different methods to converge (with "Stein" you also need to optimize the test function). More importantly, I would have liked to see results also on different data sets.   Finally, an interesting related work is "Variational Inference with Renyi Divergence", Y Li, RE Turner - arXiv preprint arXiv:1602.02311, 2016