A Probabilistic Framework for Deep Learning
We develop a probabilistic framework for deep learning based on the Deep Rendering Mixture Model (DRMM), a new generative probabilistic model that explicitly capture variations in data due to latent task nuisance variables. We demonstrate that max-sum inference in the DRMM yields an algorithm that exactly reproduces the operations in deep convolutional neural networks (DCNs), providing a first principles derivation. Our framework provides new insights into the successes and shortcomings of DCNs as well as a principled route to their improvement. DRMM training via the Expectation-Maximization (EM) algorithm is a powerful alternative to DCN back-propagation, and initial training results are promising. Classification based on the DRMM and other variants outperforms DCNs in supervised digit classification, training 2-3⇥ faster while achieving similar accuracy. Moreover, the DRMM is applicable to semi-supervised and unsupervised learning tasks, achieving results that are state-of-the-art in several categories on the MNIST benchmark and comparable to state of the art on the CIFAR10 benchmark.
The authors present a generative model, called a Rendering Model which explicitly reasons about class and nuisance variation. More formally, their model generates pixels by patch and assumes a (gaussian) noisy image pixel, and an image template that is a function of the object class and a nuisance variable. The model also makes use of a switching variable that allows the model to deal with two objects overlapping (occlusion or transparency).  They demonstrate that inference in the Rendering Model is (nearly) equivalent to a one-layer conv net which uses a batch-norm (in the beginning, rather than at the end), a fully connected or convolutional layer, a max pooling layer and a relu layer.  They combine multiple Rendering Models to create a Deep Rendering Model (DRM). The DRM is a graphical model whose layers are modified rendering models resulting in a series of affine transformations that render finer and finer details of the object as the model progresses. They argue that a relaxed form of the DRM produces a convolutional neural network.  The DRM can be trained via EM which they argue provides faster convergence than standard gradient descent.   Given their model and the drawn equivalences to convolutional neural networks, they argue that the DRM provides several insights into convolutional networks strengths and weaknesses. Firstly, since DRM inference is equivalent to max-sum message passing, a conv net is effectively doing the same thing (making a number of assumptions about its structure). Secondly, they suggest that convnets poor performance on transparent objects is due to the ambiguity of the label being assigned to the transparency. They relate this weakness to the constraint in the DRM which enforces only a single object being rendered per pixel. Thirdly, they argue that like DRMs, convnets represent appearance models for classes in a factorized-mixture form over various levels of abstraction.   They validate their method on MNIST and experiment with comparing their supervised model with a simple convnet as well as demonstrating decent results for unsupervised and semi-supervised variations of their model. They illustrate decent performances for both and very good performance using the supervised DRM.    
The paper presents a very intriguing model and make an ambitious argument about its equivalence with convolutional networks. If anything, the paper suffers slightly from an abundance of content which was very difficult (I imagine) to squeeze into the allotted space. The model is sure to be of interest to anybody working with conv nets with an interest in understanding how and why they work so well and may also generate additional future work. That being said, there were a number of questionable elements of the work which are illustrated below.  Novelty: To my knowledge, the formulation of the RM and DRM are themselves novel and the derivation (supplementary material) of the equivalence is quite interesting. This is probably the strongest part of the paper.  Technical Correctness: There is a relatively large assumption, or approximation is Proposation A.1 (a). This is not equivalent, but rather an approximation. I appreciate that the authors explain that this is in a noise-free limit of the GRM, but whether or not this is true in theory makes ALL the difference in practice. Any practical conclusions made by the paper are suspect if the assumptions made by the work are not true in practice...  Nit: the nuisance variables are 'g' in the paper but 'h' in the supplementary material. This is quite confusing.  Proposition A.1 (b) might be true for a single layer model where it has 3 channels as inputs but it will cease to be true in higher levels of the model.    There is a general claim being made between the equivalencies of the DRM and convnets. However, there is a very particular case of convents being compared to, and that is a convnet composed of (only) batch norm layers followed by conv layers, followed by max pooling layers, followed by relus. Oddly this doesn't actually match three of the more popular conv nets (AlexNet, VGG and GoogleNet) which do not use this configuration exactly (AlexNet uses local contrast normalization, VGG uses multiple convs in a row before pooling, GoogleNet uses Inception Modules which are combinations of various branches which are concatenated.    How is the image divided into patches? Stride 1 overlapping patches? How are boundary patches used?  The order of where batch-norm goes is slightly off. Most models perform batch norm AFTER the operation. Its true that in a deep model, this distinction is not meaningful (e.g. BN just goes between layers) but it certainly matters in the first and last layer.   144: overall pose g^L: this is an interpretation and needs to be validated experimentally.   Evaluation: I really wish any other dataset other than MNIST was used. This dataset has been hacked to bits and the margins in which various models operate in terms of gains or losses of performance are within a handful of samples. Consequently, I genuinely am not sure what to make of the results. That being said, their claim that the model achieves better accuracy (in the abstract) is not correct either (See DropConnect for example) and a 1-layer conv net used to compare to is a bit of a straw-man.   For a model motivated by the problem of explicitly modeling nuisance variables, it would be much better to use a dataset whose nuisance variables are clearer. What are the nuisance variables for MNIST?  Additionally, while the authors do provide a few qualitative results of the generative model (in the supplementary materials), they make a large and ambitious claim about WHAT the model is doing (providing finer and finer details of the image over each layer), but they make no attempt to validate this interpretation. Indeed, even if the DRM and DCN are equivalent in some way, they seem to have traded one problem (why do convents work so well) for another (what is each layer in the DRM really doing). For example, line 167, directly argues about the coarse-to-fine interpretation, but this needs to be validated.     Overall, the paper is quite interesting and worth discussion but there seem to be a large number of assumptions made which aren't true and the evaluation could be way better. I'm on the fence regarding acceptance.