The information bottleneck method of Tishby et al. (2000)  provides an information-theoretic method for representation learning, by training an encoder to retain all information which is relevant for predicting the label, while minimizing the amount of other, superfluous information in the representation. The original formulation, however, requires labeled data in order to identify which information is superfluous. In this work, we extend this ability to the multi-view unsupervised setting, in which two views of the same underlying entity are provided but the label in unknown. This enables us to identify superfluous information as that which is not shared by both views. A theoretical analysis leads to the definition of a new multi-view model which produces state-of-the-art results on the Sketchy dataset and on label-limited versions of the MIR-Flickr dataset. We also extend our theory to the single-view setting by taking advantage of standard data augmentation techniques, empirically showing better generalization capabilities when compared to traditional unsupervised approaches for representation learning.
This is a good multiview representation learning paper with new insights. The authors propose to learn variables z_1 and z_2, which are consistent, contain view-invariant information but discard as much view-specific information as possible.
The paper relies on mutual information estimation and is reconstruction-free. It is mentioned in some previous works (e.g. Aaron van den Oord et al. 2018), that reconstruction loss can introduce bias that has a negative effect on the learned representation.
Comparing to existing multiview representation learning approaches that try to maximize the mutual information between learned representation and the view(s), this paper clearly defines superfluous information that we should try to throw away and figure out how to obtain sufficiency learned representation for output. The authors also draw clear connections between a few existing (multiview) representation learning methods to their proposed approaches.
The experimental results on the right side of Figure 3, deliver a very interesting conclusion. In low-resource case, robust feature (obtained by using the larger beta, discarding more superfluous information) is crucial for achieving good performance. While when the amount of labeled data samples is enough, vice-versa.
Here are my major concerns:
1.	In the paper, the authors said the original formulation of IB is only applicable to supervised learning. That is true, but the variational information bottleneck paper [Alexander A. Alem et al. 2017] already showed the connection of unsupervised VIB to VAE in the appendix.
2.	I would not consider the data augmentation used to extend single-view data to “pseudo-multiview” as a contribution. This has been done before (e.g. in the multiview MNIST experiment part of the paper "On Deep Multi-View Representation Learning").
3.	Which MV-InfoMax do you really compare to? You listed a few of them: (Ji et al., 2019; Henaff et al., ´ 2019; Tian et al., 2019; Bachman et al., 2019) in the related work section.
4.	I think the authors should also make a more careful claim on their results in MIR-Flickr. 
I’d rather not saying MIB generally outperforms MV-InfoMax on MIR-Flickr, as MIB does not (clearly) outperform MV-InfoMax when enough labeled data is available for training downstream recognizers. But MIB does clearly outperform MV-InfoMax when scaling down the percentage of labeled samples used.
5.	Regarding baselines/experiments
a.	In Figure 4, it seems that VAE (with beta=4) outperforms MV-InfoMax. Why the ``"pseudo-second view" does not help Mv-Infomax in this scenario? Why VAE is clearly better than Infomax?
b.	In Figure 3, you might also tune beta for VCCA and its variants, like what you did for VAE/VIB in a single view. 
6.	Do you think your approach can be extended to more than two views easily? 
For me, it seems the extension is not trivial, as it requires o(n^2) terms in your loss for n views.
But this is minor.