Adaptive dropout for training deep neural networks
Recently, it was shown that deep neural networks can perform very well if the activities of hidden units are regularized during learning, e.g, by randomly dropping out 50% of their activities. We describe a method called 'standout' in which a binary belief network is overlaid on a neural network and is used to regularize of its hidden units by selectively setting activities to zero. This 'adaptive dropout network' can be trained jointly with the neural network by approximately computing local expectations of binary dropout variables, computing derivatives using back-propagation, and using stochastic gradient descent. Interestingly, experiments show that the learnt dropout network parameters recapitulate the neural network parameters, suggesting that a good dropout network regularizes activities according to magnitude. When evaluated on the MNIST and NORB datasets, we found that our method achieves lower classification error rates than other feature learning methods, including standard dropout, denoising auto-encoders, and restricted Boltzmann machines. For example, our method achieves 0.80% and 5.8% errors on the MNIST and NORB test sets, which is better than state-of-the-art results obtained using feature learning methods, including those that use convolutional architectures.
Summary: "Adaptive dropout for training deep 
      neural networks" describes and evaluates a new neural network 
      architecture that combines a sigmoid belief network and a conventional 
      feedforward network. Pro: * very strong results on MNIST and 
      NORB * novel model that generalizes dropout to full neural networks 
      Con: * little theoretical insight * empirical results 
      somewhat scattered * training algorithm not perfectly clear 
      Quality: The strongest element of this work, to me, is 
      the empirical results. The performance of this non-convolutional model 
      on MNIST and NORB is impressive. The intuition that the stochastic 
      units implement some Bayesian sampling over models doesn't inspire me 
      personally, but I don't object to the hypothesizing. The learning 
      algorithm appears to require more computation per iteration than 
      standard backprop and SGD, the authors should add some discussion of 
      training speed to the results / discussion. The experimental 
      results are not obviously focussed on any particular claim. For 
      example, the proposed method is a stochastic neural network, but it is 
      never simply trained as a supervised model. Why is it trained first as 
      an autoencoder? How much autoencoder training was necessary? Several 
      hyperparamters governing the optimization process appear to have been 
      tuned, were these tuned separately for the different model 
      architectures? If they weren't, then what can the reader draw from the 
      model comparison scores? For me, the experiment section mainly says 
      that using your model, Nesterov momentum, AE pre-training, and careful 
      hyperparameter optimization, you got a tweaked our fully connected 
      model to get .8 on MNIST and 5.8 on NORB. Using \alpha and \beta 
      to tie \pi to 'w' and then also using ReLU units for the 
      non-stochastic parts of the model makes it feel like there's a much 
      simpler way to explain what you've done: namely, you have used a new 
      activaction function that is a ReLU with some stochastic noise added 
      around the inflection point. This may well be a good way (like 
      maxout?) to get the learning benefits of linear units without losing 
      so many units because they just turn "off" and can't come back on 
      again. Clarity: The paper is well-organized, following 
      the standard presentation for papers of this type. "principal" 
      -> principle "each experts" -> each expert Figures 1 and 
      2 are not clear to me, what am I supposed to see? Are the images in 
      Figure 1 sorted by class label for example? Why does the leftmost one 
      look nothing like a 0 or a 1, and the rightmost one look nothing like 
      a 9 or 0. In describing the learning algorithm, could the 
      authors explain the procedure for training \pi in more detail? It 
      isn't clear to me. What is the free energy here, and how is it 
      possible to estimate each m_j's contribution fast enough for learning? 
      Section 4 is confusing, because after sketching one procedure for 
      training \pi in Section 3, this Section throws it away and makes \pi 
      an affine transformation of 'w'. It's not very good to say "we found 
      this technique works very well in practice", I'd like to see the 
      evidence for myself (that's what this paper's for). If it doesn't make 
      any difference in terms of end-of-training accuracy, then why did you 
      tie them? Why are not showing your CIFAR-10 results? In 
      line 190, you say that \alpha and \beta are "learnt" but then in line 194 
      these are described as hyper-parameters. Which is it? If 
      \pi and w are tied by \alpha and \beta, then wouldn't a single "activation 
      function" plot be a clearer way to explain the effect of \alpha and 
      \beta than Figure 3? Figure 3 is tough to interpret for me, I don't 
      know what to make of activation histograms. Originality: 
      The proposed model is novel. Significance: At 
      least some members of the NIPS community love new stochastic/neural 
      architectures that work well and no one really understands why, that 
      faction will be intrigued by this paper.