The Variational AutoEncoder (VAE) can learn simultaneously a representation and a generative model, but often, in order to generate sharper samples, a useless representation is learnt. Starting from the observation that a meaningful representation is an informative one, in this work we analyse the VAE from an information theoretic perspective. We define the concept of capacity for the variational network, and we associate the two tasks of VAE: generation and representation quality, to two information properties of the network: information of the generative model and capacity of the network. Then we suggest that an optimal generative model is the one optimising the Capacity-Constrained InfoMax (CCIM), a theoretical objective learning the maximal informative generative model while maintaining bounded the network capacity. The theoretical assumptions are confirmed by the computational experiments, where between the different families of VAE, the one optimising a variational lower bound of the CCIM generates sharper samples and learns a clustered and robust representation. We call this the Variational InfoMax AutoEncoder (VIMAE).
The paper develops an information-theoretic training scheme for Variational Auto-Encoders (VAEs). This scheme is tailored for addressing the well-known disentanglement problem of VAEs where an over-capacity encoder sometimes manages to both maximize data fit and shrink the KL divergence between the approximate posterior and prior to zero. Consequently, the latent representations of the observations become independent, making them unusable for any downstream task.
The method developed in Section 3 and proposed explicitly in Eqs 7 to 11 is novel per se, though not groundbreaking.
Figs 2 and 3 are only few examples manually chosen from rather simple tasks. In the absence of a quantitative evaluation metric, they are not informative. In the outcomes of the same runs, there might exist counterexamples where the vanilla VAE generate perceptively more appealing reconstructions than VIMAE.
As a minor comment, I think the paper unnecessarily complicates the presentation of the idea. The development of notions such as f-Divergence, links to MMD, the constrained optimization setting in Eq 5 etc., do not serve to the main story line, but only distracts a mind. I would prefer a brief intro on VAEs and directly jumping into the proposed method. Last but not least, the statement of technical novelty in Section 3 is extremely encrypted. What exactly is "absolutely" novel there and what is prior art? Can the authors give a to-the-point answer to this question during the rebuttal?
The paper has two fundamental weaknesses:
i) The paper misses a key reference, which addresses the same representation disentanglement problem using the same information-theoretic approach, only with some minor technical divergences:
Alemi et al., Fixing a Broken ELBO, ICML, 2018. 
This paper is a must-cite, plus a key baseline. This submission cannot be treated as a contribution without showing an improvement on top of this extremely closely related work. The problem is the same, the solution is almost the same, and the theoretical implications of the solution are also the same.
ii) The paper builds the entire story line around the statement that the representation disentanglement problem is caused by the properties of the ELBO formula and attempts to fix it by developing a new inference technique. This is attitude largely overlooks simpler explanations. For instance, the vanilla VAE assumes a mean field q(z|x) across the z dimensions. This makes q(z|x) fall largely apart from p(x) which definitely does not factorize that way. Earlier work has shown substantial improvements on the quality of q(z|x) when structured variational inference techniques are used, such as normalizing flows. Furthermore, there has also emerged techniques that can very tightly approximate p(x) in closed-form using the change of variables formula without performing any variational inference at all. No need to say, a tight approximation on p(x) means the same for p(z|x) simply using the Bayes rule. This leaves no room for further improvements by information-theoretic inference alternatives. For instance see:
Dinh et al., Density Estimation Using Real NVP, ICLR, 2017
The presence of such two strong alternative remedies to the problem addressed by this work makes its fundamentals shaky. The only way to get over this situation is to provide a thorough comparison against these methods, which is obviously missing.
---
Post-rebuttal: Thanks to authors for all the effort they put on the updated version. I appreciate that the paper now provides a quantitative comparison. However, my point (ii) remains fully unaddressed and I fully share the theoretical concerns raised by Reviewer 1. All in all, I keep my position on the below the threshold side with a weak reject.