Multi-Prediction Deep Boltzmann Machines
We introduce the multi-prediction deep Boltzmann machine (MP-DBM). The MP-DBM can be seen as a single probabilistic model trained to maximize a variational approximation to the generalized pseudolikelihood, or as a family of recurrent nets that share parameters and approximately solve different inference problems. Prior methods of training DBMs either do not perform well on classification tasks or require an initial learning pass that trains the DBM greedily, one layer at a time. The MP-DBM does not require greedy layerwise pretraining, and outperforms the standard DBM at classification, classification with missing inputs, and mean field prediction tasks.
Deep Boltzmann Machines (DBNs) are usually 
      initialized by greedily training a stack of RBMs, and then fine-tuning the 
      overall model using persistent contrastive divergence (PCD). To perform 
      classification, one typically provides the mean-field features to a 
      separate classifier (e.g. a MLP) which is trained discriminatively. 
      Therefore the overall process is somewhat ad-hoc, consisting of L + 2 
      models (where L is the number of hidden layers) each with its own 
      objective. This paper presents a holistic training procedure for DBNs 
      which has a single training stage (where both input and output variables 
      are predicted) producing models which can classify directly as well as 
      efficiently performing other tasks such as imputing missing inputs. The 
      main technical contribution is the mechanism by which training is 
      performed; a way of training DBNs which uses the mean field equations for 
      the DBN to induce recurrent nets that are trained to solve different 
      inference tasks (essentially predicting different subsets of observed 
      variables). Quality: I think this is a really interesting paper 
      and its implications to DBN training are important. I agree with the 
      authors that, despite DBNs being really powerful models, the generally 
      accepted way of training them has several shortcomings. The authors 
      explicitly point these out in section 3: greedily training is sub-optimal, 
      using multiple task-specific models loses some benefit of probabilistic 
      modeling, and the greedy training procedure is overly complex. The 
      motivation is strong, the methods are presented clearly, the work is 
      intuitive and clever, and the experiments demonstrate that the method 
      works (even if the only datasets considered are the old MNIST and small 
      NORB). Clarity: Generally the paper is well written. There are far 
      too many top-level sections (11)! Combine 4- 8 into Methods, and 9-10 into 
      Experiments. Also, even after reading the caption for figure 2 c) several 
      times, I still didn't fully understand it. Originality: To the 
      best of my knowledge, the approach is original (an earlier version 
      appeared at ICLR workshop track). Significance: The paper is 
      important to the deep learning/representation learning community. In fact, 
      by simplifying the DBN training procedure, the paper may encourage more 
      DBN activity specifically those who don't have experience with layer-wise 
      training procedures (as the authors point out in section 1). I wouldn't be 
      surprised if this work doesn't lead to more subset-like training 
      procedures for other architectures. The multi-inference "trick" described 
      in section 5 is interesting in its own right (separate from the DBN 
      application). Comments- Even though "y" is almost 
      universally used for output and "x" input, the second paragraph in section 
      refers to these variables without defining them. Please define these 
      variables before using them.- With regards to not being able to 
      train a larger MP-DBM on NORB (and thus validate the effectiveness of 
      holistic prediction on more than 1 dataset), there is no reason not to try 
      distributing the model over multiple GPUs. As well, recent work 
      "Predicting Parameters in Deep Learning" (Denil et al. Arxiv) gives 
      another way to reduce effective model size but retain capacity without 
      having to go to distributed models.