In this paper, we develop an unsupervised generative clustering framework that combines variational information bottleneck and the Gaussian Mixture Model. Specifically, in our approach we use the variational information bottleneck method and model the latent space as a mixture of Gaussians. We derive a bound on the cost function of our model that generalizes the evidence lower bound (ELBO); and provide a variational inference type algorithm that allows to compute it. In the algorithm, the coders' mappings are parametrized using neural networks and the bound is approximated by Markov sampling and optimized with stochastic gradient descent. Numerical results on real datasets are provided to support the efficiency of our method.
The author(s) posit a Mixture of Gaussian's prior for a compressed latent space representation of high-dimensional data (e.g. images and documents). They propose fitting this model using the Variational Information Bottleneck paradigm and explicate its derivation and tie it to the variational objective used by similar models. They empirically showcase their model and optimization methodology on the MNIST, STL-10, and Reuters10k benchmarks.
The idea of using a latent mixture of Gaussian's to variationally encode high-dimensional data is not new. The author(s) appropriately cite VaDE (Jiang, 2017) and DEC (Xie, 2016), "Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders" (Dilokthanakul, 2017). However, the author(s) unfortunately did not mention "Approximate Inference for Deep Latent Gaussian Mixtures" (Nalisnick, 2016). Nalisnick et al. consider the exact generative process as the proposed by the author(s), but with the addition of a Dirichlet prior on the Categorical distribution. Nalisnick et al. fit their model with a VAE and circumvent Dirichlet intractability by positing a Kumarswamy stick-breaking variational posterior (the resulting distribution is on the simplex). They achieve 91.58% accuracy, but do so after fitting a KNN to the latent space. New techniques such as "Implicit Reparameterization Gradients" (Figurnov, 2018) and "Pathwise Derivatives Beyond the Reparameterization Trick" (Jankowiak, 2018) allow direct use of a Dirichlet posterior in VAEs. These methods are respectively implemented in the TensorFlow and PyTorch APIs. My point here is that there are many ways to fit this pre-existing model. From my review of these other works we have, for "best run" on MNIST, that author(s) > VaDE > Nalisnick > Dilokthanakul > DEC. Thus, the author(s) are SoTA for this particular generative process for their best run. It would be nice to see their standard deviation to gauge how statistically significant their results are. However, I am dubious of the SoTA claim as I detail later.
The author(s) derivation was sound, but a bit confusing for me. In particular, I found keeping track of P's and Q's very burdensome after reading sections 2.1 and 2.2. In my experience, Q is typically reserved for a variational distribution that approximates some intractable P distribution, while P is used to describe the generative process (i.e. likelihood) and other exact distributions. Furthermore, one typically introduces the generative process first using P distributions. Once, I got past this confusion everything else made sense. I might suggest introducing the generative process first and with P distributions instead of Q's. The variational model can follow with the Q distributions as the author(s) have it. Equations 4, 5, and 6 all exactly match the unsupervised information bottleneck objective (Alemi, 2017)--see appendix B. I am therefore confident in those equations. I carefully checked their derivation of the VADe comparison (equation 10 and appendix B). Their derivation is straightforward. The principle trick is using the MC assumption C->X->U for P distributions to claim p(u|x,c) = p(u|x). Equations 12-16 all follow naturally from equation 11. If equation 17 is indeed an approximation or bound approximation, I would suggest not using the equals sign. Instead, consider another appropriate operator or rename Dkl to indicate it is the approximated version (just as in equation 24).
If the author(s) like my suggestion regarding generative vs variational nomenclature, I would also change the second sentence of page 6 to something like "We use our variational Qc|u distribution to compute assignments." Thereafter, I would drop the star indicator for optimal parameters. These parameters are not necessarily optimal given the non-convexity of the DNN. Replace with, "after convergence, we ..." Similarly, drop optimal from line 2 of algorithm 1.
Circling back to the experiments, the author(s) use reported values from DEC and VaDE. Those works compute cluster assignments using a KNN classifier on the latent space. This paper however uses the arg max of equation 19. I much prefer this article's method, but for comparison purposes, the author(s) should similarly use a KNN classifier on their latent space to compute accuracy in the same manner. The use of KNN in these other works allows them to consider a number of latent clusters larger than the number of true classes (e.g. 20 clusters for MNIST). I like that the author(s) stick to 10 clusters for MNIST, but for comparison I would have liked to see a KNN generated accuracy alongside their equation 19 based accuracy. It looks like the author(s) implemented VADe for STL-10 based on table 1. If so, it seems they could easily implement equation 19 for their STL-10 value for VADe. If not, please correct this. Not to belabor further, but I would really like to see a table 2 that reports both equation 19 and KNN accuracies when available. Namely, the author(s) should report both values for their model and can leave equation 19 accuracies blank for reported values.
Lastly, I always raise an eyebrow when I see tSNE latent space representations. STL-10 was used to generate figure 6, where one needs dim(u) >> 2. However, MNIST can be well reconstructed using just 2 latent dimensions. In this case, tSNE is unnecessary. The author(s) state "Colors are used to distinguish between clusters." This statement is unclear as to whether the author(s) are using the class label or learned latent cluster. If it is the former, figure 6 makes sense in that it shows misclassifications (i.e. red dots in the green cluster). However, if it is the latter then I am concerned tSNE is doing something weird.
To summarize, I enjoyed the paper. My only concern is novelty. Being the first to pair an existing model with an existing method, in my eyes, does not necessarily meet the ICLR bar. The author(s) seemingly achieve SoTA, but without KNN-based accuracies for their model it is hard to compare to cited works. Having these KNN results and error bars would strengthen their case substantially to: achieving SoTA for an existing model by being the first to pair it with an existing method.