The joint optimization of representation learning and clustering in the embedding space has experienced a breakthrough in recent years. In spite of the advance, clustering with representation learning has been limited to flat-level categories, which oftentimes involves cohesive clustering with a focus on instance relations. To overcome the limitations of flat clustering, we introduce hierarchically clustered representation learning (HCRL), which simultaneously optimizes representation learning and hierarchical clustering in the embedding space. Specifically, we place a nonparametric Bayesian prior on embeddings to handle dynamic mixture hierarchies under the variational autoencoder framework, and to adopt the generative process of a hierarchical-versioned Gaussian mixture model. Compared with a few prior works focusing on unifying representation learning and hierarchical clustering, HCRL is the first model to consider a generation of deep embeddings from every component of the hierarchy, not just leaf components. This generation process enables more meaningful separations and mergers of clusters via branches in a hierarchy. In addition to obtaining hierarchically clustered embeddings, we can reconstruct data by the various abstraction levels, infer the intrinsic hierarchical structure, and learn the level-proportion features. We conducted evaluations with image and text domains, and our quantitative analyses showed competent likelihoods and the best accuracies compared with the baselines.
This paper proposes using a variant of the nested CRP as a prior on the latent space of a variational autoencoder. The authors demonstrate that this approach is able to simultaneously learn a meaningful latent representation of high-dimensional data (text and images) and do hierarchical clustering in that space.
Pros:
* The high-level idea is compelling.
* The empirical results are compelling, and the evaluation is thorough.
Cons:
* The prose is pretty rough. The paper is full of sentences like "VAE-nCRP trade-off is the direct dependency modeling among clusters against the mean-field variational approach" that don't convey their intended meaning (at least to me).
* The random variable η seems completely superfluous. It only affects the likelihood through the level indicator l, but the marginal distribution p(l) = ∫_η p(η, l)dη \propto α is tractable, since only one level is drawn per observation. (This is not the case for the traditional nCRP as used in topic modeling, since there a different level is chosen for each word.)
* The novelty over the nCRP-VAE approach of Goyal et al. (2017) is pretty minor. The main difference seems to be that the model can select clusters at different levels, but I didn't quite get the intuition for why this should be desirable. In topic modeling, higher-level clusters tend to contain less-specialized words, and each document is a mix of specialized and general topics. But in this model, only one level is used to explain an entire image or document, and the idea that an entire image or document is much "more specialized" than another doesn't seem very intuitive to me.