Factoring Variations in Natural Images with Deep Gaussian Mixture Models
Generative models can be seen as the swiss army knives of machine learning, as many problems can be written probabilistically in terms of the distribution of the data, including prediction, reconstruction, imputation and simulation. One of the most promising directions for unsupervised learning may lie in Deep Learning methods, given their success in supervised learning. However, one of the current problems with deep unsupervised learning methods, is that they often are harder to scale. As a result there are some easier, more scalable shallow methods, such as the Gaussian Mixture Model and the Student-t Mixture Model, that remain surprisingly competitive. In this paper we propose a new scalable deep generative model for images, called the Deep Gaussian Mixture Model, that is a straightforward but powerful generalization of GMMs to multiple layers. The parametrization of a Deep GMM allows it to efficiently capture products of variations in natural images. We propose a new EM-based algorithm that scales well to large datasets, and we show that both the Expectation and the Maximization steps can easily be distributed over multiple machines. In our density estimation experiments we show that deeper GMM architectures generalize better than more shallow ones, with results in the same ballpark as the state of the art.
This paper introduces the idea of deep Gaussian mixture models.  A GMM can be seen as consisting of a single isotropic unit norm Gaussian, where each of the components of the mixture consists of applying a different linear transformation to that Gaussian.  This idea is extended to the case of a multilayer network, where each node in the network corresponds to a linear transformation, and each route through the network corresponds to a sequence of linear transformations.  The number of mixture components is then the number of routes through the network.This is a clearly written paper.  The idea is both simple and good.  Overall I liked the paper.  I did think that the experimental results were disappointingly low dimensional, especially considering how much effort was spent in the paper discussing how to make the algorithm computationally tractable.Also, I think the proposed algorithm is quite closely related toTang, Yichuan, Ruslan Salakhutdinov, and Geoffrey Hinton. "Deep mixtures of factor analysers." International Conference on Machine Learning (2012).this probably deserves a citation and discussion.I would strongly encourage the authors to release the source code for the experimental results as supplemental material.  This makes the paper more convincing, increases citation counts, and encourages a culture of reproducible science which benefits everyone.More detailed comments follow:39 - "in function" -> "as a function"73 - "A_3,1" -> "A_1,3"89 - You provide good arguments in 312-318 for why this is true for your algorithm.  I'm skeptical that EM is more parallelizable than stochastic gradient descent in general though.  Maybe phrase this more specifically.139 - There are an exponential number of paths through the network.  Assigning a probability to each of them without factorizing them would become intractable quite quickly -- even for relatively small networks.  I'm surprised this factorization isn't needed almost all the time.199 - This is commonly called the MAP approximation.  Why not instead just sample a single nonzero gamma with probability proportional to pi (this is EM, with the E step represented as a sample from the posterior).  Unlike when using the MAP approximation, this will give you an unbiased estimate of your learning gradient, and should lead to a higher log likelihood model.230 - "in function of" -> "as a function of"235 - In the plot it never converges to the optimum.251 - "and scalable." -> "and in a scalable way."307, 341 - If you want to do quasi-Newton optimization with minibatches, and without hyperparameters, you might try https://github.com/Sohl-Dickstein/Sum-of-Functions-Optimizer312-318 - Cool.  This provides good support for the benefits of EM.331 - "have to construction" -> "have to construct"337 - with using -> using344 - I strongly suspect this is a side effect of the MAP approximation to the posterior, as opposed to not optimizing fully during the M step.  (As a note -- EM can be seen as maximizing a lower bound on the log likelihood.  Importantly, that lower bound increases even if the M step is not run to convergence.)Figure 4 - I don't think this X-axis makes much sense for the Deep GMMs -- there's nothing special about the number of components in the top layer.  Number of parameters would probably make a better x-axis.436 - Author name listed as "anonymous".