Kullback-Leibler Proximal Variational Inference
We propose a new variational inference method based on a proximal framework that uses the Kullback-Leibler (KL) divergence as the proximal term. We make two contributions towards exploiting the geometry and structure of the variational bound. First, we propose a KL proximal-point algorithm and show its equivalence to variational inference with natural gradients (e.g., stochastic variational inference). Second, we use the proximal framework to derive efficient variational algorithms for non-conjugate models. We propose a splitting procedure to separate non-conjugate terms from conjugate ones. We linearize the non-conjugate terms to obtain subproblems that admit a closed-form solution. Overall, our approach converts inference in a non-conjugate model to subproblems that involve inference in well-known conjugate models. We show that our method is applicable to a wide variety of models and can result in computationally efficient algorithms. Applications to real-world datasets show comparable performances to existing methods.
The paper presents a general method for non-conjugate variational inference based on proximal method and linearisation of the non-conjugate model. This is shown to reduce to natural gradient optimisation for conjugate exponential models. The method is shown to lead to slightly better predictive accuracy than standard approximate inference methods in a few selected problems and data sets.      Quality      The method relies on linearisation to handle non-conjugate models. The seems potentially problematic, as previous works have found linearisation to be unreliable in variational inference with non-conjugate models (see e.g. Honkela and Valpola, NIPS 2004). The method is evaluated empirically on a few data sets, and mostly found to perform well. The paper focuses a lot on the positive aspects of the method, and its weaknesses and limitations are not mentioned at all.      Clarity      The paper is mostly clearly written and well-organised, but some key details are missing (specifically: what exactly is f_n).      Originality      The proposed approach is novel, although it is based on combination of existing techniques from various fields. Use of references is a bit shaky: the good fundamental reference from proximal algorithms seems to be missing and furthermore previous approaches using linearisation with non-conjugate variational inference are not discussed.      Significance      As the reliability of the proposed method is questionable, it is difficult to judge the paper's importance until this is resolved.      Other comments      1. The last sentence of the abstract contains unsubstantiated advertising: you method is not the best in every way. This must be clarified or deleted.      2. In Eqs. (2)-(3), what is eta?      3. In Eq. (3), why is it arg min and not arg max as in Eq. (2)?      4. As the proposed method seems to be a purely batch algorithm, it is unclear why it cites and links to SVI methods so much. Classic batch VB would seem much more relevant here.      5. You should clarify what f_n is precisely in the examples you study. Also, the accuracy of the linearisation procedure needs to be checked to make sure you do not suffer from similar problems as illustrated in Fig. 1 of Honkela and Valpola (NIPS 2004) for the Taylor series linearisation.      Additional references      Previous use of linearisation in variational inference:     Lappalainen, Harri, and Antti Honkela. "Bayesian non-linear independent component analysis by multi-layer perceptrons." In Advances in independent component analysis, pp. 93-121. Springer London, 2000.      Analysis of problems with linearisation:     Honkela, Antti, and Harri Valpola. "Unsupervised variational Bayesian learning of nonlinear models." In Advances in neural information processing systems, pp. 593-600. 2004.