We introduce the Mutual Information Machine (MIM), an autoencoder framework for learning joint distributions over observations and latent states. The model formulation reflects two key design principles: 1) symmetry, to encourage the encoder and decoder to learn different factorizations of the same underlying distribution; and 2) mutual information, to encourage the learning of useful representations for downstream tasks. The objective comprises the Jensen-Shannon divergence between the encoding and decoding joint distributions, plus a mutual information regularizer. We show that this can be bounded by a tractable cross-entropy loss between the true model and a parameterized approximation, and relate this to maximum likelihood estimation and variational autoencoders. Experiments show that MIM is capable of learning a latent representation with high mutual information, and good unsupervised clustering, while providing NLL comparable to VAE (with a sufficiently expressive architecture).
The authors present the Mutual Information Machine (MIM), which is essentially a  latent variable model learned via Adversarially Learned Inference with regularizations that prefer learning representations with high mutual information. While the method is interesting, it does not seem to be well justified in representation learning, and the empirical results are not well-justified due to the use of weak baselines.
Pros:
The idea of formulating the L_MIM objective appears interesting, which uses the mixture of P_\theta and Q_\theta and minimizes an upper bound to H(x) + H(z) - I(x, z), so it is effectively minimizing a lower bound to I(x, z), under the proposed symmetric distribution.
Cons:
Symmetry argument:
	- Prior work on maximizing mutual information for latent variables focus on the MI as evaluated in the "encoder distribution", whereas in this paper the argument is focused on symmetry, i.e. the MI estimated is based on \mathcal{M}_s. 
	- However, for practical purposes, it is unclear how this is superior than the "encoder distribution" paradigm. Eventually, we wish to use the z for some downstream tasks, which requires us to obtain z from some distribution (e.g. the encoder q(z|x)) and then run classification on this z. 
	- Unless we sample z | x from the conditional distribution of \mathcal{M}_s as proposed by the authors, it is unclear why we need this symmetry in representation learning; it seems slower to sample from the MIM distribution though (see motivation for A-MIM).
	- The authors did not demonstrate results that compare A-MIM and MIM on the same architecture.
Failure to discuss certain related literature, such as Alemi et al. 2017 and Zhao et al. 2018. These are also "strongly related to VAEs" and related to mutual information objectives with latent variables.
	- Alemi et al. 2017 discussed this in the context of "rate discussion trade-off"
	- Zhao et al. 2018 proposed objectives to find specific trade-offs under constraints
In fact, Zhao et al. 2018 considered the MIs in both P_theta and Q_theta, and one should be able to obtain a "symmetric" MI by adding these two Mis uniformly (although again, I am not sure how this is useful for downstream applications on the model).
Using VAE as a baseline comparison. 
	- It is well known that the VAE objective does not encourage maximizing mutual information, and various existing works have already been proposed to address this problem. Outperforming VAE in terms of MI is hence not very surprising.
	- Methods that encourage mutual information maximization under the latent variable generative modeling framework would be more suited for comparison. How does MIM compare with the methods introduced in Chen et al. 2016, Alemi et al. 2017 and Zhao et al. 2018? 
	- In the context of ALI and BiGAN, one could also apply other approaches like MINE / InfoMAX to maximize MI. Again, there is no comparison with these approaches. 
	- Alemi and Zhao both showed that there is a trade-off between consistency and mutual information under the same architecture (using their objectives). It seems that what MIM empirically does is also to sacrifice NLL for better MI, and not clear whether MIM gives comparable or better trade-offs.
The paper would be more convincing if the authors empirically compare with other sensible baselines and justify why we should use MIM as opposed to A-MIM in representation learning.
[relevant papers]
Chen et al. 2016 Lossy VAEs
Alemi et al. 2017 Fixing a broken ELBO
Zhao et al. 2018 A Lagrangian Perspective to Latent Variable Generative Models