Matrix Manifold Optimization for Gaussian Mixtures
We take a new look at parameter estimation for Gaussian Mixture Model (GMMs). Specifically, we advance Riemannian manifold optimization (on the manifold of positive definite matrices) as a potential replacement for Expectation Maximization (EM), which has been the de facto standard for decades. An out-of-the-box invocation of Riemannian optimization, however, fails spectacularly: it obtains the same solution as EM, but vastly slower. Building on intuition from geometric convexity, we propose a simple reformulation that has remarkable consequences: it makes Riemannian optimization not only match EM (a nontrivial result on its own, given the poor record nonlinear programming has had against EM), but also outperform it in many settings. To bring our ideas to fruition, we develop a welltuned Riemannian LBFGS method that proves superior to known competing methods (e.g., Riemannian conjugate gradient). We hope that our results encourage a wider consideration of manifold optimization in machine learning and statistics.
This paper revisits fitting mixtures of Gaussians with gradient methods. A combination of manifold optimization, with a particular linesearch procedure, and a "reparameterized" objective, empirically beats batch EM in the experiments reported.While I'm not an expert in Riemannian manifolds, the work appears plausible. The work is a reported at a level that's possible to follow, if taking results from cited references on trust.Plain mixtures of Gaussians may not seem particularly exciting in this world of Bayesian nonparametrics and deep neural networks. However, GMMs are quietly plodding along as important work-horses in applications: they're useful and surprisingly powerful. People often then want to extend GMMs in various ways, e.g., by adding constraints or tying covariances together. But then EM usually doesn't apply. Gradient-based procedures would generalize fine, so ones that work well on the base model could have significant impact on statistical applications.I would previously have used an unconstrained parameterization based on the Cholesky. While the paper gives reasons to disfavour this approach, it's a shame it's not included in the empirical comparison. Are spurious stationary points an issue in practice? Is the manifold optimizer actually better? Is the main gain here from the "representation" in (2.3); would that also help the Cholesky approach? The paper would be stronger if it answered these questions.  Minor comments:Line 081: "open a new" -> "open new"Line 138: rely *on* geodesicsThe timings in Table 4 would be probably be better as a graph. A graph would give an idea of the comparison more quickly. Precise times are irrelevant, being implementation and machine dependent.I suggest replacing lines 147-149 "At any point... specifically [4]" with: "Geodesics on the manifold from $\Sigma_1$ to $\Sigma_2$ are given by [4]:". Riemannian metrics are not explained in the paper, and don't need to be understood or mentioned to take on trust the geodesic result. (The lack of context, and the nearby definition of `$d$', encourges a misreading of $d$ as a variable, making the trace expression evaluate to `$d^3$' (!).)I suggest adding the zero mean ("0,") explicitly to the normal density in (2.3) and the line below it. On a quick reading, augmenting the data with a constant in the line above (2.3) seems like a terrible idea, a Gaussian could fit these new data with infinite density. The zero-mean constraint is what stops that from happening. Also, the compact Gaussian notation is only defined explicitly including a mean term above (2.1).I wouldn't call the transformed problem (2.3) a "reparameterization". A different model is being fitted, one that would generate data from a different distribution than the original.I suggest numbering all equations so other people can refer to them, not just in reviews, but in reading groups and citations.