Published as a conference paper at ICLR 2018 KERNEL IMPLICIT VARIATIONAL INFERENCE
Recent progress in variational inference has paid much attention to the flexibility of variational posteriors. One promising direction is to use implicit distributions, i.e., distributions without tractable densities as the variational posterior. However, existing methods on implicit posteriors still face challenges of noisy estimation and computational infeasibility when applied to models with high-dimensional latent variables. In this paper, we present a new approach named Kernel Implicit Variational Inference that addresses these challenges. As far as we know, for the first time implicit variational inference is successfully applied to Bayesian neural networks, which shows promising results on both regression and classification tasks.
This paper presents Kernel Implicit Variational Inference (KIVI), a novel class of implicit variational distributions. KIVI relies on a kernel approximation to directly estimate the density ratio. Importantly, the optimal kernel approximation in KIVI has closed-form solution, which allows for faster training since it avoids gradient ascent steps that may soon get "outdated" as the optimization over the variational distribution runs. The paper presents experiments on a variety of scenarios to show the performance of KIVI.
Up to my knowledge, the idea of estimating the density ratio using kernels is novel. I found it interesting, specially since there is a closed-form solution for this estimate. The closed form solution involves a matrix inversion, but this shouldn't be an issue, as the matrix size is controlled by the number of samples, which is a parameter that the practitioner can choose. I also found interesting the implicit MMNN architecture proposed in Section 4.
The experiments seem convincing too, although I believe the paper could probably be improved by comparing with other implicit VI methods, such as [Liu & Feng], [Tran et al.], or others.
My major criticism with the paper is the quality of the writing. I found quite a few errors in every page, which significantly affects readability. I strongly encourage the authors to carefully review the entire paper and search for typos, grammatical errors, unclear sentences, etc.
Please find below some further comments broken down by section.
Section 1: In the introduction, it is unclear to me what "protect these models" means. Also, in the second paragraph, the authors talk about "often leads to biased inference". The concept to "biased inference" is unclear. Finally, the sentence "the variational posterior we get in this way does not admit a tractable likelihood" makes no sense to me; how can a posterior admit (or not admit) a likelihood?
Section 3: The first paragraph of the KIVI section is also unclear to me. In Section 3.1, it looks like the cost function L(\hat(r)) is different from the loss in Eq. 1, so it should have a different notation. In Eq. 4, I found it confusing whether L(r)=J(r). Also, it would be nice to include a brief description of why the expectation in Eq. 4 is taken w.r.t. p(z) instead of q(z), for those readers who are less familiar with [Kanamori et al.]. Finally, the motivation behind the "reverse ratio trick" was unclear to me (the trick is clear, but I didn't fully understand why it's needed).
Section 4: The first paragraph of the example can be improved with a brief discussion of why the methods of [Mescheder et al.] and [Song et al.] "are nor applicable". Also, the paragraph above Eq. 11 ("When modeling a matrix...") was unclear to me.
Section 6: In Figure 1(a), I think there must be something wrong, because it is well-known that VI tends to cover one of the modes of the posterior only due to the form of the KL divergence (in contrast to EP, which should look like the curve in the figure). Additionally, Figure 3(a) (and the explanation in the text) was unclear to me. Finally, I disagree with the discussion regarding overfitting in Figure 3(b): that plot doesn't show overfitting because it is a plot of the training loss (and overfitting occurs on test); instead it looks like an optimization issue that makes the bound decrease.
**** EDITS AFTER AUTHORS' REBUTTAL ****
I increased the rating to 7 after reading the revised version.