Altitude Training: Strong Bounds for Single-Layer Dropout
Dropout training, originally designed for deep neural networks, has been successful on high-dimensional single-layer natural language tasks. This paper proposes a theoretical explanation for this phenomenon: we show that, under a generative Poisson topic model with long documents, dropout training improves the exponent in the generalization bound for empirical risk minimization. Dropout achieves this gain much like a marathon runner who practices at altitude: once a classifier learns to perform reasonably well on training examples that have been artificially corrupted by dropout, it will do very well on the uncorrupted test set. We also show that, under similar conditions, dropout preserves the Bayes decision boundary and should therefore induce minimal bias in high dimensions.
-------------------------------------------review adjustment based on authors feedback-------------------------------------------In their response to the original review, the authors corrected my mistake and indeed the derivation is correct. Therefore, they have removed my main concern about this work. There are some issues that are left unclear and I hope the authors have a chance to clarify them. In (1) for example, the behavior of the RHS as a function of \delta is unclear. In their response, the authors said that this formula is correct only for \delta=1/2. Therefore, the discussion about the bias-variance tradeoff cannot be driven by this formula.------------------Original comments------------------Dropout seems to be an important component of the recent success of deep neural nets. This study attempt to shed some light on the reasons for this success. This line of research is of significance if it will allow us to come up with better regularization techniques. The authors study dropouts in the context of topic modeling where the data is assumed to be generated by a specific process. The main contribution of this work is a theoretical analysis that suggest that dropouts should increase accuracy. The paper is well written and contains novel contributions. The topic is relevant to the NIPS community. Unfortunately, there is a main component of the analysis that I fail to understand. In page 4, line 204 it is claimed that the variance after dropouts is (1-\delta) times the original variance (the same argument is repeated in page 11 line 593 of the appendix). However, IMHO, the variance is (1-\delta)^2 times the original variance. This is so since if X and Y are random variables such that Y= bX  where b is a scalar than the Var(Y)=b^2 Var(x). Am I wrong about it? Is it possible to fix the proof despite this issue?Another way to see this problem is looking at Figure 1b. It is implied from this figure that if X~N(\mu,\sigma^2) then the probability P[X < 0] is smaller than the probability P[Y < 0] where Y=(1-\delta)X. However, it is clear that these probabilities are identical as long as (1-\delta) > 0. I think that the presentation can be improved in (1) and (9). Following these formulas, in line 63 and in line 192 it says "the constants in the bound depend only on \delta". If a constant depend on \delta it is probably not a constant. This presentation requires the discussion in line 64 that explains why the bound does not behave as you may expect when \delta is close to one. Instead, making the dependency on \delta more explicit may be useful. One way to do it, that will keep the formula readable is to add a constant \Kappa to the R.H.S. in (1) and (9) and define \Kappa in a separate equation.In line 88 it is claimed that delta allows control over the bias-variance tradeoff. However, in line 81 it is claimed that the dropout does not introduce much of a bias. These claims seem to contradict each other. It would be helpful to clarify this issue.The theoretical analysis studies the zero-one loss but the experiments use the logistic loss (line 117). It would be nice to have some arguments to support these choices.In line 170 there seems to be a typo in the text: "the error on the original measure is actually much smaller than the error on the original measure"In (7) and (8) it is not clear if this is the training error or test error. In section 6 (Experiments and Discussion) there is no statistical significance analysis of the results. The plots don't contain error bars and therefore it is hard to tell if the results are artifact of randomness or do they represent a real phenomenon.