We present an approach for efficiently training Gaussian Mixture Models (GMMs) with Stochastic Gradient Descent (SGD) on large amounts of high-dimensional data (e.g., images). In such a scenario, SGD is strongly superior in terms of execution time and memory usage, although it is conceptually more complex than the traditional Expectation-Maximization (EM) algorithm. For enabling SGD training, we propose three novel ideas: First, we show that minimizing an upper bound to the GMM log likelihood instead of the full one is feasible and numerically much more stable way in high-dimensional spaces. Secondly, we propose a new annealing procedure that prevents SGD from converging to pathological local minima. We also propose an SGD-compatible simplification to the full GMM model based on local principal directions, which avoids excessive memory use in highdimensional spaces due to quadratic growth of covariance matrices. Experiments on several standard image datasets show the validity of our approach, and we provide a publicly available TensorFlow implementation.
The paper proposes a new method based on Stochastic Gradient Descent to train Gaussian Mixture Models, and studies this method especially in the context of high dimension. 
The method is based on optimizing max-lower bound to the log-likelihood, together with a regularization term, using stochastic gradient descent. 
The method is applied to two image datasets, and seem to produce sensible results. 
However, in my opinion, the results and presentation do not seem at the level suitable for publication.
There are no guarantees for convergence/other performance measures for the new method, in contrast to recent methods based on moment matching (e.g Ge, Huand and Kakade, 2015). Therefore, the new method and paper should provide excellent empirical results to be worthy of publication. 
The authors write in the 'related work' section that GMM with regularization was proposed by [Verbeek et al. 2005], but it is an older idea - for example [Ormoneit&Tresp 1998] 
In Section 3.1, the motivation for the maximization in eq. (2) is unclear. 
Why is it easier to compute the gradient this way rather than keep the original likelihood?
Moreover, the max operation is non-smooth and can cause problems with the definition of the gradient at some points. 
The authors point to a problem of underflow/overflow when evaluating the gradient of the 
full log-likelihood because the densities p(x | k) can be very small - but it is standard practice in EM to keep all probabilities multiplied by say their maximum p_max and keep log(p_max) separately, to avoid underflow problems. 
Section 3.2: I don't understand the requirement \Sigma_{i,j} >=0. Is it a requirement for each entry of the covariance matrix? (which covariance matrix? there are K such matrices). The requirement should be that each matrix is positive-definite, not the entries.
Section 3.3: The first sentence is wrong - EM can also suffer from the problem of local minima. 
Also, the single-component solution doesn't seem like a local minimum - but rather the log-likelihood is unbounded here
since you can put another component on one of the data points with infinitely small variance. 
The degenerate solution where all Gaussians have equal weights, mean and variance does not seem like a local minimum
of the log-likelihood. Say the data really comes from 2 distinct Gaussians - then separating the two Gaussians means a bit would increase the log-likelihood. is not a local minimum. I'm not even sure if the gradient is zero at this point - the authors should show this. Maybe the authors mean their modified loss L_MC - this should be stated clearly. 
The change of regularization during the training process seems like a heuristic that worked well for the authors, but it is thus unclear what optimization problem is the optimization solving. The regularization is thus included for optimization reasons, and not in the usual sense of regularization. 
The expression for \tau at the end of Section 3.3 seems wrong to me. I don't see how plugging it into eq. (7) gives a continuous \sigma(t) function. 
Section 3.4: What is \mu^i? I didn't see a definition 
I don't understand the local principal directions covariance structure. The authors write 'a diagonal covariance matrix of S < D entries). But what about the other D-S coordinates? are they all zero? or can have any values? 
The parameters count lists (S+1)*D+1 for each Gaussian so I'm assuming S*D parameters are used for the covariance matrix, but it is unclear how. Eq. (9) has the parameters vectors d_{ks} for the principal directions, together with the \Sigma_ss scalar values - it would be good to relate them to the mean and variance of the Gaussians. 
Section 4: The paragraph that describes the experimental details at the beginning is repeated twice. 
The experimental results are not very convincing. The images in Figure 2,4 were picked by an unknown (manual?) criteria. 
In the comparison to EM in Figure 3 there are missing details - which log-likelihood is used? L, L_MC? or different ones for different methods? is this test set log-likelihood? what fraction of the data was used? 
There is also no comparison of running time between the two methods.