Hierarchical Methods of Moments
Spectral methods of moments provide a powerful tool for learning the parameters of latent variable models. Despite their theoretical appeal, the applicability of these methods to real data is still limited due to a lack of robustness to model misspecification. In this paper we present a hierarchical approach to methods of moments to circumvent such limitations. Our method is based on replacing the tensor decomposition step used in previous algorithms with approximate joint diagonalization. Experiments on topic modeling show that our method outperforms previous tensor decomposition methods in terms of speed and model quality.
obtained by standalone EM in this case is reasonable and desirable: when asked for a small number of latent variables EM yields a model which is easy to interpret and can be useful for data visualization and exploration. For example, an important application of low-dimensional learning can be found in mixture models, where latent class assignments provided by a simple model can be used to split the training data into disjoint datasets to which EM is applied recursively to produce a hierarchical clustering [12,13]. The tree produced by such clusterings procedure provides a useful aid in data exploration and visualization even if the models learned at each branching point do not accurately represent the training data.
In this paper we develop a hierarchical method of moments that produces meaningful results even in misspecified settings. Our approach is different from previous attemps to design MoM algorithms for misspecified models. Instead of looking for convex relaxations of existing MoM algorithms like in [14][15][16] or analyzing the behavior of a MoM algorithm with a misspecified number of latent states like in [17,18], we generalize well-known simultaneous diagonalization approaches to tensor decomposition by phrasing the problem as a non-convex optimization problem. Despite its non-convexity, the hierarchical nature of our method allows for a fast accurate solution based on low-dimensional grid search. We test our method on synthetic and real-world datasets on the topic modeling task, showcasing the advantages of our approach and obtaining meaningful results.
Spectral methods for learning latent variable models reduce the learning problem to the problem of performing some low-rank factorization over matrices collecting some moments of the target distribution. There are many variants of this in the literature, most approaches use SVD or other forms of tensor decompositions. This paper highlights an important limitation of many of these approaches. The main limitation is that most theoretical guarantees apply when the goal is to recover a model of k states assuming that the mixture distribution from which the moment matrix is computed also has k components. However, in many cases one is interested in learning models for l lower than k, what the authors call the misspecified setting.
This paper proposes a method that performs joint diagonalization by combining a
truncated SVD step with an optimisation. The theoretical analysis shows that this
algorithm can work in the misspecified setting. The paper presents an application
of the proposed algorithm to topic modelling.
I believe this is a nice contribution, it is true that there are already many
variants of the spectral method, each with its own claimed advantages. This been
said, the one proposed in this paper is technically sound and addresses a true
limitation of previous work.
In terms of references the paper is quite complete, there are however a couple of
relevant missing references: 
- B.Balle et al. "Local Loss Optimisation in operator models: A new insight into spectral learning"; since it is one of the first papers to frame spectral learning directly as an optimization. 
- More closely related
to this submission, B. Balle et al "A Spectral Learning of Finite State
Transducers" is to the best of my knowledge the first paper that proposes a
spectral method based on joint diagonalization.