Improved Dropout for Shallow and Deep Learning
Dropout has been witnessed with great success in training deep neural networks by independently zeroing out the outputs of neurons at random. It has also received a surge of interest for shallow learning, e.g., logistic regression. However, the independent sampling for dropout could be suboptimal for the sake of convergence. In this paper, we propose to use multinomial sampling for dropout, i.e., sampling features or neurons according to a multinomial distribution with different probabilities for different features/neurons. To exhibit the optimal dropout probabilities, we analyze the shallow learning with multinomial dropout and establish the risk bound for stochastic optimization. By minimizing a sampling dependent factor in the risk bound, we obtain a distribution-dependent dropout with sampling probabilities dependent on the second order statistics of the data distribution. To tackle the issue of evolving distribution of neurons in deep learning, we propose an efficient adaptive dropout (named evolutional dropout) that computes the sampling probabilities on-the-fly from a mini-batch of examples. Empirical studies on several benchmark datasets demonstrate that the proposed dropouts achieve not only much faster convergence and but also a smaller testing error than the standard dropout. For example, on the CIFAR-100 data, the evolutional dropout achieves relative improvements over 10% on the prediction performance and over 50% on the convergence speed compared to the standard dropout.
This paper proposes a new dropout scheme based on sampling features or neurons according to a multinomial distribution with different probabilities for different features or neurons.  For shallow learning, it proposes a data-dependent dropout by using the second-order statistics of the features to compute the sampling probabilities.  Theoretical analysis of the risk bound is provided.  For deep learning, it proposes an evolutional dropout by using the second-order statistics of each layer’s output based on a mini-batch of examples (for the purpose of reducing the computational demand) to compute the sampling probabilities of dropout for that layer.  Some experiments are presented to compare the proposed distribution-dependent dropout with standard dropout for both shallow and deep learning and with batch normalization. 
TECHNICAL QUALITY  It is discussed in Section 1 (lines 31-33) that features with low/zero variance can be dropped more frequently or even completely.  How is this intuition supported by the theoretical analysis in Section 4 (particularly Eq. (8) or (9))?  Note that the features are not automatically zero-meaned.  The uniform dropout scheme described in line 135 (as a special case of multinomial dropout when all the sampling probabilities are equal) is only similar but not identical to standard dropout (the sampling probabilities for different features are not i.i.d.).  It may not be good to use it in the experiments as if it is indeed the standard dropout scheme.  There are some other concerns about the experiments reported.  Although the emphasis of this paper is not on obtaining better prediction performance by trying different network structures and tricks (line 265-266), it aims at improving standard dropout and hence is natural to also include other improved dropout schemes as baselines in the comparative study.  Such baselines should at least include the adaptive dropout method in Ba and Frey’s NIPS 2013 paper (which should have been cited) and the variational dropout method by Kingma et al. [6].  The experiments consider only one dropout rate (0.5).  Different dropout rates should be reported to give a more complete picture.  For shallow learning, I suggest that the author also includes a setting for s-dropout using data after Z-normalization.  As noted in the remark in Section 4.1 (lines 213-217), its performance is expected to be similar to that of d-dropout.  In Section 5.3, the paper concludes that e-dropout is roughly a randomized version of batch normalization plus standard dropout.  This seems to imply that combining e-dropout with batch normalization will not lead to further improvement in performance.  I suggest that this combination be also included in the comparison.  Also, from Figure 2 (right), as the number of iterations increases the test accuracy of e-dropout fluctuates more than other methods, including BN+dropout which, like e-dropout, is also a randomized scheme.  This seems to show that e-dropout is slightly worse than BN+dropout as far as learning stability is concerned.  Some discussions should be provided on this.  The performance gap between s-dropout and e-dropout is not always as large as that between s-dropout (without Z-normalization) and d-dropout.  In terms of the final test accuracy, e-dropout does not always win.  In fact, it seems to be worse than s-dropout for MNIST.  NOVELTY  The main theoretical result in the paper is Theorem 1.  Its proof is based on standard techniques for stochastic gradient descent.  The extension from shallow learning to deep learning is straightforward in that it simply uses mini-batches instead of the full data set to establish the sampling probabilities for each layer separately.  No theoretical guarantee is available for the deep learning case.  Nevertheless, as far as I know, this is the first data-dependent dropout method proposed with theoretical justification though only for the shallow learning case.  IMPACT  From the experiments, the performance gap between s-dropout and e-dropout is not always as large as that between s-dropout and d-dropout.  One may question whether the improvement obtained by distribution-dependent dropout will still be significant for deeper networks.  Compared with more realistic deep learning applications which need much deeper networks, I am afraid only small-scale experiments are presented for deep learning in the paper.  Consequently, the potential impact that this work can bring to the deep learning community is unclear.  CLARITY & PRESENTATION  The paper is generally well organized and easy to read.  Nevertheless, the writing has room to improve.  Besides problems with English usage, there are also some language/formatting errors.  Here are some examples: L22-23: “at random samples neurons and sets their outputs to be zeros” L32: “can dropout” L58: “acts similar to” L67: “a faster convergence” L73: “reminder” L80,85: seem to be errors when using citation in LaTeX L86: “to speed-up” L93-94: “developed evolutional dropout” L101: “in next section” L111: “where he operator” L135: “we refer as” L167-168: “a non-convex optimization” L172: “does not loss the generality” L179: “upper bond” L181: “The detailed proof of theorem” L185-186: “the second term also depend on” L191: “included in supplement”