LEARNING LATENT SUPERSTRUCTURES IN VARIA- TIONAL AUTOENCODERS FOR DEEP MULTIDIMEN- SIONAL CLUSTERING
We investigate a variant of variational autoencoders where there is a superstructure of discrete latent variables on top of the latent features. In general, our superstructure is a tree structure of multiple super latent variables and it is automatically learned from data. When there is only one latent variable in the superstructure, our model reduces to one that assumes the latent features to be generated from a Gaussian mixture model. We call our model the latent tree variational autoencoder (LTVAE). Whereas previous deep learning methods for clustering produce only one partition of data, LTVAE produces multiple partitions of data, each being given by one super latent variable. This is desirable because high dimensional data usually have many different natural facets and can be meaningfully partitioned in multiple ways.
This paper introduces a new VAE model, the latent tree VAE (LTVAE), which aims to learn models with multifaceted clustering, that is separate clusterings are enforced on different subsets of the latent features.  This is achieved using a tree-structured prior on a set of discrete "super latent variables" (Y_1,...,Y_L) that identify which cluster the datapoint falls into for each separate facet (i.e. there is a separate clustering associated with each Y_n).  The subset of the standard latent variables z then form a Gaussian mixture model (GMM) for each Y_n.   Both the structure of this setup (i.e. the associated graphical model) and the parameters (i.e. means and variances of the clusters) are learned during training.  This introduces a number of computational challenges not usually seen in for VAE training, for which, seemingly well thought through, novel schemes are introduced, most notably a message passing scheme for calculating gradients of the log marginal p(z).
Overall I think this is a very good paper.  The exposition of the work is, for the most part, very good - the paper was a pleasure to read.  I think that the key idea is novel and adds something unique and useful to the literature, I thus think it is work which will be of substantial interest to the ICLR community.  The quality of the paper is also very good: algorithmic details seem to have been well thought through and the experimental evaluation is above average, both in terms of apparent performance and in the breadth of experiments considered.  I would very much like to see this work accepted to ICLR and I think that the extra use of space over 8 pages in the submission is justified.  However, I do have some questions and concerns that I would like to see addressed in the rebuttal period and I may lower my score if they are not.
The key issues I would like to see addressed further discussion on are:
a) There is no discussion about what is done for the encoder in the paper.  This is surely a very important consideration here as if the encoder is not expressive enough, this will impact the learned models.  For example, the dependency structures of the latent space induce particular dependencies in the posterior that must be carefully handled to avoid harming the learning (see e.g. https://arxiv.org/abs/1712.00287).
b) I would like to see some numerical results for the similarity between the different clusterings that are learned.  A lot of the novelty of the work rests on being able to pick up different clusterings with the different facets.  However, the results suggest that the clusterings may actually have very significant overlap and so this should be quantified.
c) The approach is presuming substantially slower than a setup where the structure is pre-fixed.  I think it is fine even if there is a big slow down, but I would like to see timing information so that the reader can assess how much higher the time cost is.
d) As far as I can tell (sorry if I have made a mistake), the presented results are from single runs.  I would like to see information about the variability across different runs so that the fragility of the approach can be assessed.
e) I would like to see more justification for having a dependency structure between the Y's, ideally both in motivating this choice and in experimental evaluation to check it (more generally ablation studies for different components of the algorithm would improve the paper).  Might it be possible to use this in a way the encourages the different clusterings to be distinct from one another?
Other comments:
1) Though the writing is generally very good, there are a few exceptions:
- The second paragraph in the intro becomes a list of related work from the point where DEC is introduced.  This should be moved to the related work to improve the flow (just cite those papers at the end of the first sentence in the third paragraph) and it would be good for it to be less of a list of separate things and more something that puts the current work in the context of other approaches.
- The paragraph after Eq 3 needs some rewriting
- The explanations around and including equations 5 and 6 were quite poor: \pi is referred to but not used, it is not made clear that that g is the gradient of log p(z) instead of p(z), use brackets for the log in Eq 6 to avoid ambiguity
2) The reference formatting is wrong (i.e. cite is used everywhere instead of citep)
3) I thought the motivation for the approach in the intro was very good
4) As the seemingly most related work, it would be good to elaborate more on the Goyal et al paper and the differences of your approach to theirs.  Is there a reason this is not used as a baseline in the experiments?
5) I could not understand the step from the gradient to the gradient of the log in Eq 6.  Is this because p(y_b|z) = f(y_b) Norm(..)?
6) The text in figures 2 and 3 is too small and difficult to make out.
7) I think it is misleading to talk about p(z) as being a marginal likelihood and would use the term marginal prior, or just marginal, instead.
8) I thought Figure 4b provided a nice demonstration.
9) Is there a reason that log likelihood / ELBO scores are only provided on MNIST and only for the LTVAE / VAE?  I might be wrong, but I thought at least some of the other baselines provide this and those results presumably already exist as a side effect from calculating the clustering scores?  Relatedly, I'm aware that a previous version of this work included estimates of the normalized mutual information -- is there any reason these are no longer included?
10) Did the larger dimensional latent spaced used for the qualitative results improve or worsen the performance of previous metrics?
Minor points / typos
- mehod -> method
- of generation network -> of the generation network
- brackets in eq 7
- MoG not defined in section 4.5