Amortized inference has led to efficient approximate inference for large datasets. The quality of posterior inference is largely determined by two factors: a) the ability of the variational distribution to model the true posterior and b) the capacity of the recognition network to generalize inference over all datapoints. We analyze approximate inference in variational autoencoders in terms of these factors. We find that suboptimal inference is often due to amortizing inference rather than the limited complexity of the approximating distribution. We show that this is due partly to the generator learning to accommodate the choice of approximation. Furthermore, we show that the parameters used to increase the expressiveness of the approximation play a role in generalizing inference rather than simply improving the complexity of the approximation.
=======
Update:
The new version addresses some of my concerns. I think this paper is still pretty borderline, but I increased my rating to a 6.
=======
This article examines the two sources of loose bounds in variational autoencoders, which the authors term “approximation error” (slack due to using a limited variational family) and “amortization error” (slack due to the inference network not finding the optimal member of that family).
The existence of amortization error is often ignored in the literature, but (as the authors point out) it is not negligible. It has been pointed out before in various ways, however:
* Hjelm et al. (2015; https://arxiv.org/pdf/1511.06382.pdf) observe it for directed belief networks (admittedly a different model class).
* The ladder VAE paper by Sonderby et al. (2016, https://arxiv.org/pdf/1602.02282.pdf) uses an architecture that reduces the work that the encoder network needs to do, without increasing the expressiveness of the variational approximation. That this approach works well implies that amortization error cannot be ignored.
* The structured VAE paper by Johnson et al. (2016, https://arxiv.org/abs/1603.06277) also proposes an architecture that reduces the load on the inference network.
* The very recent paper by Krishnan et al. (posted to arXiv days before the ICLR deadline, although a workshop version was presented at the NIPS AABI workshop last year; http://approximateinference.org/2016/accepted/KrishnanHoffman2016.pdf) examines amortization error as a core cause of training failures in VAEs. They also observe that the gap persists at test time, although it does not examine how it relates to approximation error.
Since these earlier results existed, and approximation-amortization decomposition is fairly simple (although important!), the main contributions of this paper are the empirical studies. I will try to summarize the main novel (i.e., not present elsewhere in the literature) results of these:
Section 5.1:
Inference networks with FFG approximations can produce qualitatively embarrassing approximations.
Section 5.2:
When trained on a small dataset, training amortization error becomes negligible. I found this surprising, since it’s not at all clear why dataset size should lead to “strong inference”. It seems like a more likely explanation is that the decoder doesn’t have to work as hard to memorize the training set, so it has some extra freedom to make the true posterior look more like a FFG.
Also, I think it’s a bit of an exaggeration to call a gap of 2.71 nats “much tighter” than a gap of 3.01 nats.
Section 5.3:
Amortization error is an important contributor to the slack in the ELBO on MNIST, and the dominant contributor on the more complicated Fashion MNIST dataset. (This is totally consistent with Krishnan et al.’s finding that eliminating amortization error gave a bigger improvement for more complex datasets than for MNIST.)
Section 5.4:
Using a restricted variational family causes the decoder to learn to induce posteriors that are easier to approximate with that variational family. This idea has been around for a long time (although I’m having a hard time coming up with a reference).
These results are interesting, but given the empirical nature of this paper I would have liked to see results on more interesting datasets (Celeb-A, CIFAR-10, really anything but MNIST). Also, it seems as though none of the full-dataset MNIST models have been trained to convergence, which makes it a bit difficult to interpret some results.
A few more specific comments:
2.2.1: The \cdot seems extraneous to me.
5.1: What dataset/model was this experiment done on?
Figure 3: This can be inferred from the text (I think), but I had to remind myself that “IW train” and “IW test” refer only to the evaluation procedure, not the training procedure. It might be good to emphasize that you don’t train on the IWAE bound in any experiments.
Table 2: It would be good to see standard errors on these numbers; they may be quite high given that they’re only evaluated on 100 examples.
“We can quantitatively determine how close the posterior is to a FFG distribution by comparing the Optimal FFG bound and the Optimal Flow bound.”: Why not just compare the optimal with the AIS evaluation? If you trust the AIS estimate, then the result will be the actual KL divergence between the FFG and the true posterior.