We consider the problem of information compression from high dimensional data. Where many studies consider the problem of compression by non-invertible transformations, we emphasize the importance of invertible compression. We introduce new class of likelihood-based Auto-Encoders with pseudo bijective architecture, which we call Pseudo Invertible Encoders. We provide the theoretical explanation of their principles. We evaluate Gaussian Pseudo Invertible Encoder on MNIST, where our model outperforms WAE and VAE in sharpness of the generated images.
PIE extend NICE and Real NVP into situations which require having a smaller dimensionality of the latent variable (d) compared to the dimensionality of the observed variable (D), i.e. d < D. This is done by learning an extension function g(z) from R^d to R^{D-d} and then using the change of variables formula on x and [z, g(z)]. To model probabilistically the deterministic function g(z) is replaced by Normal distribution with mean g(z) and a small variance.
PIE is used to build deep generative models and trained on the MNIST dataset. The authors show that the models learnt via PIE produce sharper samples than VAEs and Wasserstein autoencoders (WAEs). No comparison to real NVP is made, which should be the main baseline of comparison to answer the question of "what is the advantage of having d < D?". Further MNIST is no longer a good enough benchmark to evaluate deep generative models. Most representative work in this literature use CIFAR-10, downsampled Imagenet, or Imagenet at 256x256.
This work falls short of the standards of ICLR in a few ways:
1. The presentation is unclear. The explanation of the extension-restriction idea is overly complicated. Further, the paper does very little to properly contextualize this work in the literature. Real NVP and flow-based models are mentioned but the proposed technique is not compared to it. The authors say they "introduce new class of likelihood-based Auto-Encoders", but this is false as far as I understood. The technique is not even an autoencoder since a separate decoder is not trained, and is obtained by exactly inverting the encoder as in real NVP.
2. The experiments are weak. The samples shown are of poor quality, and on a very simplisitic dataset (MNIST). The authors compare with vanilla VAEs, but ignore more recent improvements to VAE such as VAE-IAF, flow-based models, and also autoregressive models. A heuristic is used to measure sharpness and only used to compare against VAE and WAE. Since all these models allow likelihood evaluation, likelihoods should also have been compared.
3. The technique itself is a small change over real NVP and it's not clear whether this change brings any improvements or provides any insights about generative modeling.