Coupled Variational Bayes via Optimization Embedding
Variational inference plays a vital role in learning graphical models, especially on large-scale datasets. Much of its success depends on a proper choice of auxiliary distribution class for posterior approximation. However, how to pursue an auxiliary distribution class that achieves both good approximation ability and computation efficiency remains a core challenge. In this paper, we proposed coupled variational Bayes which exploits the primal-dual view of the ELBO with the variational distribution class generated by an optimization procedure, which is termed optimization embedding. This flexible function class couples the variational distribution with the original parameters in the graphical models, allowing end-to-end learning of the graphical models by back-propagation through the variational distribution. Theoretically, we establish an interesting connection to gradient flow and demonstrate the extreme flexibility of this implicit distribution family in the limit sense. Empirically, we demonstrate the effectiveness of the proposed method on multiple graphical models with either continuous or discrete latent variables comparing to state-of-the-art methods.
The paper proposes a new distribution class that can be used as an auxiliary distribution class for variational inference. This new class is called 'Optimization embedding' and is derived based on two steps: first a primal-dual rewriting of ELBO, and second, and optimization procedure based on the minor descent algorithm. a Particular instance of this distribution class is described that can be trained using gradient-decent techniques. The model is evaluated among three axis: the generative ability of the approach, the flexibility of the resulting approximation posterior distribution and sample efficiency. These experiments are made on classical datasets (including toy datasets, MNIST and CelebA) and show that the proposed class has good properties.   First of all, the model presented in the paper is interesting and is a good contribution to the field.  The paper is very technical and very difficult to understand for a non-expert. It needs a strong background on different methods (e.g normalizing flows, Fenchel-duality and interchangeability principle, mirror descent algorithm, etc...) An effort could be put on briefly describing these different concepts to allow the reader to better understand the big picture of the paper. But the paper is quite well written, and the steps to obtain the final distribution class are clear enough. To help the reader, I would advise the authors to include some parts of the supplementary material directly in the paper, particularly Algorithm 2 and Section B that provide a concrete example of the use of the presented principles. The first paragraph of Section 3 would also benefit from presenting the intuitive ideas behind the obtained distribution class, and particularly the role of the embeddings in the resulting algorithm. Theorem 1 seems to have some typos problems (R^p instead of R^r), and the role of v^* is not clear in this theorem. Section 3.2 is particularly difficult to understand and would benefit from a small description of the MDA principles. The experimental results are interesting and demonstrate the power of the contribution. Of course, a quantitative study on the quality of the generated samples could help to validate the generative ability of the model (using mechanical turk for example). Moreover, the convergence speed is evaluated in term of epochs, while it would be more interesting to evaluate it in term of time since the complexity of each epoch is not clearly stated in the paper, so it is not clear if the proposed approach is really faster than baselines.    Summary: interesting contribution, but a paper lacking a clear presentation of the main ideas, making it difficult to read for non-experts, and thus restricted to a small audience.