Discovering Structure in High-Dimensional Data Through Correlation Explanation
We introduce a method to learn a hierarchy of successively more abstract representations of complex data based on optimizing an information-theoretic objective. Intuitively, the optimization searches for a set of latent factors that best explain the correlations in the data as measured by multivariate mutual information. The method is unsupervised, requires no model assumptions, and scales linearly with the number of variables which makes it an attractive approach for very high dimensional systems. We demonstrate that Correlation Explanation (CorEx) automatically discovers meaningful structure for data from diverse sources including personality tests, DNA, and human language.
This paper presents an unsupervised dimensionality reduction algorithm which is based on the information bottleneck (IB) method. The method optimizes a constrained objective which, like the IB method, is comprised of the mutual information criteria. The criteria are between a joint density of discrete observed variables and the densities of a set of discrete latent factors, and between factored densities of the observed variables and the latent factors' densities. The goal of the the objective function is to infer latent factors such that by conditioning on them, the observed variables can be factorized into subsets of minimally correlated elements. An iterative algorithm is derived for optimizing the objective which alternates between updating conditional densities over the latent factors given a set of observed training data (using fixed-point equations) and updating the assignments of the observed variables to one of the latent factors. In the numerical experiments the model is applied to learn latent tree structures on a number of benchmarks. The hierarchical structures are learned by stacking multiple instances of the model such that maximum a posteriori latent interpretations of a trained model become the observed variables for the next model instance on top.The paper presents a simple method for unsupervised dimensionality reduction which scales linearly w.r.t. the number and cardinality of variables involved. The objective function it optimizes allows for an automatic selection of the number of latent factors to be employed. Experimental results on synthetic data as well as on benchmarks from different domains show that the stacked implementation of the method performs a decent job of learning plausible/close to ground-truth latent hierarchies as compared to the other methods. On the other hand, the work lacks details about the consistency of the proposed method. The model uses fixed-point updates for estimating posteriors over the latents which are set randomly at the beginning. Moreover, the values of alpha_{i,j} are also randomly initialized (violating constraints on them) and are then gradually updated using lambda and adaptive gamma_{i,j} parameters. To that end however, no analysis is performed on model's convergence behaviour e.g., for increasingly large number and cardinalities of model variables. Also details are lacking on the sampling of training data and it is not clear how much a chosen sampling regime may effect the quality of probability estimates and in turn overall parameter evolution and performance. The method is proposed for discrete random variables and it also imposes the condition that each of the observed variables belongs to one latent factor, hence claiming that the model requires no assumptions is not actually true.For the relevant work, how does the proposed method can be contrasted with the recent work of Song et al. ICML 2013, on hierarchical decomposition of latent tree structures?Altogether the paper is well written and it is clear about conveying the main message and results.