Dropout Training as Adaptive Regularization
Dropout and other feature noising schemes control overfitting by artificially corrupting the training data. For generalized linear models, dropout performs a form of adaptive regularization. Using this viewpoint, we show that the dropout regularizer is first-order equivalent to an L 2 regularizer applied after scaling the features by an estimate of the inverse diagonal Fisher information matrix. We also establish a connection to AdaGrad, an online learning algorithm, and find that a close relative of AdaGrad operates by repeatedly solving linear dropout-regularized problems. By casting dropout as regularization, we develop a natural semi-supervised algorithm that uses unlabeled data to create a better adaptive regularizer. We apply this idea to document classification tasks, and show that it consistently boosts the performance of dropout training, improving on state-of-the-art results on the IMDB reviews dataset.
*Summary of the paper: This paper studies 
      "dropout training" in the framework of feature noising as a 
      regularization. It is know that adding an additive gaussian noise 
      to the feature is equivalent to an l_2 regularization in a least square 
      problem (Bishop). This paper studies multiplicative Bernoulli 
      feature noising, in a shallow learning architecture, with a general loss 
      function and shows that it has the effect of adapting the geometry through 
      an "l_2 regularizer " that rescales the feature (beta^{\top} D(beta,X) 
      beta). The Matrix D(beta,X) is a estimate of the inverse diagonal 
      fisher information. It is worth noting that D does not depend on the 
      labels. The equivalent regularizer of dropout is non convex in general. 
      A connexion to AdaGrad in online learning introduced in Duchi et 
      al is made, as both approaches reduces to adapting the geometry towards 
      rare but predictive features. The Matrix D(beta,X) could be 
      estimated using unlabeled data, authors devise a semi supervised variant 
      of the equivalent dropout regularizer. Comments: - 
      Dropout as originally introduced by hinton et al considers multiplicative 
      Bernoulli noising for deep learning architectures (neural nets). An 
      average neural net is obtained, by training independently many neural nets 
      where some hidden layers were dropped out at random. Hinton's dropout 
      seems to be close to ensemble methods (boosting, bagging etc.), and a 
      different analysis is required to understand this paradigm even with one 
      layer. Referring to Hinton's dropout in the introduction is a bit 
      misleading as there is no models averaging in the current paper. It 
      would be more precise to mention that this work consider another form of 
      dropout, where we learn one model robust to Bernoulli multiplicative 
      perturbation. - The regularizer is not an l_2 regularizer as D 
      depends on beta also in a non linear way through the hessian of the 
      likelihood. one should be careful with this appellation. - It is 
      interesting to derive the approximate equivalent regularizer, to 
      understand the effect of dropout. The non convexity of the regularizer 
      poses computational issues. Authors mention in a footnote that they use 
      lbfgs, more explanation and discussion are needed to understand how to 
      avoid local minimas. The non convexity of the equivalent regularizer 
      is not discussed in section 6. - The parameter (delta/1-delta) 
      corresponds to a regularization parameter. any insight on how to choose 
      delta? according to Hinton et al , delta=1/2 seems to have a good 
      performance. - The log partition function is sometimes hard to get 
      in closed from any insight on how to go around that? Authors 
      answered promptly questions raised by the reviewer.