PREVENTING POSTERIOR COLLAPSE WITH δ-VAES
Due to the phenomenon of "posterior collapse," current latent variable generative models pose a challenging design choice that either weakens the capacity of the decoder or requires altering the training objective. We develop an alternative that utilizes the most powerful generative models as decoders, optimize the variational lower bound, and ensures that the latent variables preserve and encode useful information. Our proposed δ-VAEs achieve this by constraining the variational family for the posterior to have a minimum distance to the prior. For sequential latent variable models, our approach resembles the classic representation learning approach of slow feature analysis. We demonstrate our method's efficacy at modeling text on LM1B and modeling images: learning representations, improving sample quality, and achieving state of the art log-likelihood on CIFAR-10 and ImageNet 32 × 32.
General:
The paper attacks a problem of the posterior collapse that is one of the main issues encountered in deep generative models like VAEs. The idea of the paper relies on introducing a constraint on the family of variational posteriors in such a way that the KL term could be controlled.
The authors propose to use a linear autoregressive process (AR(1)) as the prior. Alternatively, they trained a single-layer LSTM network with conditional-Gaussian outputs as the prior (the auxiliary prior). Additionally, the authors claim that the encoder should contain anti-causal dependencies in order to introduce additional bias that may diminish the posterior collapse.
The experiments present various results on image and text datasets. Interestingly, the proposed techniques allowed to perform on a par with purely autoregressive models, however, the latent variables were utilized (i.e., no posterior collapse). For instance, in Figure 3(a) we can notice that a decoder is capable of generating similar images for given latent variable. A similar situation is obtained for text data (e.g., Figure 12).
In general, I find the paper interesting and I believe it should be discussed during ICLR.
Pros:
+ The paper is well-written and all ideas are clearly presented.
+ The idea of “hard-coded” constraints is interesting and constitutes an alternative approach to utilizing either quantized values in the VAE (VQ-VAE) or a constrained family of variational posteriors (e.g., Hyperspherical VAE).
+ The obtained results are convincing. Additionally, I would like to highlight that at the first glance it might seem that there is no improvement over the autoregressive models. However, the proposed approach allows to encode an image or a document and then decode it. This is not a case for purely autoregressive models.
+ The introduction of the Slow Features into the VAE framework constitutes an interesting direction for future research.
Cons:
- The quality of Figure 4 is too low.
- I am not fully convinced that the auxiliary prior is significantly better than the AR(1) prior. Indeed, the samples seem to be a bit better for the aux. prior but it is rather hard to notice by inspecting quantitative metrics.
- In general, the proposed approach is a specific solution rather than a general framework. Nevertheless, I find it very interesting with a potential for future work.