Variational autoencoders (VAE), (Kingma & Welling, 2013;Rezende et al., 2014), learn probabilistic latent variable models by optimizing a bound on the marginal likelihood of the observed data. Beyond providing a good density model a VAE model assigns to each data instance a latent code. In many applications, this latent code provides a useful high-level summary of the observation. However, the VAE may fail to learn a useful representation when the decoder family is very expressive. This is because maximum likelihood does not explicitly encourage useful representations and the latent variable is used only if it helps model the marginal distribution. This makes representation learning with VAEs unreliable. To address this issue, we propose a method for explicitly controlling the amount of information stored in the latent code. Our method can learn codes ranging from independent to nearly deterministic, while benefiting from decoder capacity. Thus, we decouple the choice of decoder capacity and the latent code dimensionality from the amount of information stored in the code.
Summary
This paper proposes a penalized VAE training objection for the purpose of increasing the information between the data x and latent code z.  Ideally, optimization would consist of maximizing log p(x) - | I(x,z) - M |, where M is the user-specified target mutual information (MI) and I(x,z) is the model’s current MI value, but I(x,z) is intractable, necessitating the use of an auxiliary model r(z|x).  Optimization, then, consists of alternating gradient ascent on the VAE parameters and r’s parameters.  Experiments on simulations and text data are reported, showing that increasing M has the desired effect of allowing more deviation from the prior.  Specifically, this is shown through text generation where the sampled sentences become more varied as M is decreased and better reconstructed as M is increased.  
Evaluation
Pros:  I like how this paper formalizes failure in representation learning as information loss in z---although the formulation is not particularly novel, i.e. [Zhao et al., ArXiv 2017]), and constructs an explicit, penalized objective to allow the user to specify the amount of information retained in z.  In my opinion, the proposed objective is more transparent than the objectives proposed by related work.  For instance, Chen et al.’s (2017) Lossy VAE, while aiming to solve essentially the same problem, does so by parameterizing the prior and using a windowed decoder, but there is no explicit control mechanism as far as I’m aware (except for how many parameters / window size).  Perhaps the Beta-VAE’s [Higgins et al., ICLR 2017] KLD weight is similarly interpretable (as beta increases, less information is retained), but I like that M has the clear meaning of mutual information---whereas the beta in the Beta-VAE is just a Lagrangian.  In terms of experiments, I like the first simulation; it’s a convincing sanity check.  As for the second, I like the spirit of it, but I have some criticisms, as I’ll explain below.
Cons:  The method requires training an auxiliary model r(z|x) to estimate I(x,z).  While I don’t find the introduction of r(z|x) problematic, I do wish there was more discussion and analysis of how well the mutual information is being approximated during training, especially given some of the simplifying assumptions, such as r(z|x)=p(z|x).  If the MI estimate is way off, that detracts from the method and makes an alternative like the Beta-VAE---which doesn’t require an auxiliary model---more appealing, since what makes the MAE superior---its principled targeting of MI---does not hold in practice.
As for the movie review experiment, I find the sentence samples a bit anecdotal.  Was the seed sentence (“there are many great scenes of course”) randomly chosen or hand picked?  Was this interpolation behavior typical?  I ask these questions because I find the plot in Figure 3 all but meaningless.  It’s good that we see reconstruction quality go up as M increases, as expected, but the baseline VAE model is a strawman.  How does reconstruction percentage look for the Bowman et al. (2015) VAE?  What about the Beta-VAE?  Or Lossy VAE?  Figure 3 would be okay if there were more experiments, but as it is the only quantitative result, more work should have gone in to it.  For instance, a compelling result would be if we see one or more of the models above plateau in reconstruction percentage and the MAE surpass that plateau.
Conclusions
While I found aspects of this paper interesting, I recommend rejection primarily for two reasons.  The first is that I would like to see how well the mutual information is being estimated during training.  If the estimate is way off, this makes the method less appealing as what I like about it---the interpretable MI target---is not really a ‘target’, in practice, and rather, is a rough hyperparameter similar to the Beta-VAE’s beta term (which has the added benefit of no auxiliary model).  The second reason is the paper’s weak experimental section.  The only quantitative result is Figure 3, and while it shows reconstruction percentage increases with M, there is no way to contextualize the number as the only comparison model is a weak VAE, which gives ~ 0%.  Questions I would like to see answered: How good is the MI estimate?  How close is the converged VAE to the target?  How does the model compare to the Bowman et al. VAE or the Beta-VAE?  (It would be quite compelling to show similar or improved performance without the training tricks used by Bowman et al.)  Can we somehow estimate the appropriate M directly from data (such as based on the entropy of training or validation set) in order to set the target rigorously?         
1.  S. Zhao, J. Song, and S. Ermon.  “InfoVAE: Information Maximizing Variational Autoencoders.”  ArXiv 2017.
2.  X. Chen, D. Kingma, T. Salimans, Y. Duan, P. Dhariwal, J. Shulman, I. Sutskever, and P. Abbeel.  “Variational Lossy Autoencoder.”  ICLR 2017.
3.  I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerchner. “Beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework.”  ICLR 2017
4.  S. Bowman, L. Vilnis, O. Vinyas,  A. Dai, R. Jozefowicz, and S. Bengio.  “Generating Sentences from a Continuous Space.”  CoNLL 2016.