Dropout with Expectation-linear Regularization
Dropout, a simple and effective way to train deep neural networks, has led to a number of impressive empirical successes and spawned many recent theoretical investigations. However, the gap between dropout’s training and inference phases, introduced due to tractability considerations, has largely remained under-appreciated. In this work, we first formulate dropout as a tractable approximation of some latent variable model, leading to a clean view of parameter sharing and enabling further theoretical analysis. Then, we introduce (approximate) expectation-linear dropout neural networks, whose inference gap we are able to formally characterize. Algorithmically, we show that our proposed measure of the inference gap can be used to regularize the standard dropout training objective, resulting in an explicit control of the gap. Our method is as simple and efficient as standard dropout. We further prove the upper bounds on the loss in accuracy due to expectation-linearization, describe classes of input distributions that expectation-linearize easily. Experiments on three image classification benchmark datasets demonstrate that reducing the inference gap can indeed improve the performance consistently.
This paper puts forward a not entirely new, but also not sufficiently understood interpretation of dropout regularization. The authors derive useful theorems that estimate or put bounds on key quantities that are of interest when analyzing dropout regularized networks from their perspective. They furthermore introduce an explicit regularization term that should have a well understood impact on these key quantities. In the experimental section they convincingly show that the proposed regularization indeed has the expected effect and that their perspective on dropout is therefore useful and meaningful. Their proposed regularization also seems to have a positive impact on the models performance but they demonstrate this only on rel. small scale benchmark problems. I therefore don’t belief that this approach will have a large impact on how practitioner train models. But their general perspective is well aligned with the recently proposed idea of “Dropout as a bayesian approximation” and the insights and theorems in this paper might enable future work in that direction.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Dropout with Expectation-linear Regularization
Dropout, a simple and effective way to train deep neural networks, has led to a number of impressive empirical successes and spawned many recent theoretical investigations. However, the gap between dropout’s training and inference phases, introduced due to tractability considerations, has largely remained under-appreciated. In this work, we first formulate dropout as a tractable approximation of some latent variable model, leading to a clean view of parameter sharing and enabling further theoretical analysis. Then, we introduce (approximate) expectation-linear dropout neural networks, whose inference gap we are able to formally characterize. Algorithmically, we show that our proposed measure of the inference gap can be used to regularize the standard dropout training objective, resulting in an explicit control of the gap. Our method is as simple and efficient as standard dropout. We further prove the upper bounds on the loss in accuracy due to expectation-linearization, describe classes of input distributions that expectation-linearize easily. Experiments on three image classification benchmark datasets demonstrate that reducing the inference gap can indeed improve the performance consistently.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper puts forward a not entirely new, but also not sufficiently understood interpretation of dropout regularization. The authors derive useful theorems that estimate or put bounds on key quantities that are of interest when analyzing dropout regularized networks from their perspective. They furthermore introduce an explicit regularization term that should have a well understood impact on these key quantities. In the experimental section they convincingly show that the proposed regularization indeed has the expected effect and that their perspective on dropout is therefore useful and meaningful. Their proposed regularization also seems to have a positive impact on the models performance but they demonstrate this only on rel. small scale benchmark problems. I therefore don’t belief that this approach will have a large impact on how practitioner train models. But their general perspective is well aligned with the recently proposed idea of “Dropout as a bayesian approximation” and the insights and theorems in this paper might enable future work in that direction.
assistant
This paper puts forward a not entirely new, but also not sufficiently understood interpretation of dropout regularization. The authors derive useful theorems that estimate or put bounds on key quantities that are of interest when analyzing dropout regularized networks from their perspective. They furthermore introduce an explicit regularization term that should have a well understood impact on these key quantities. In the experimental section they convincingly show that the proposed regularization indeed has the expected effect and that their perspective on dropout is therefore useful and meaningful. Their proposed regularization also seems to have a positive impact on the models performance but they demonstrate this only on rel. small scale benchmark problems. I therefore don’t belief that this approach will have a large impact on how practitioner train models. But their general perspective is well aligned with the recently proposed idea of “Dropout as a bayesian approximation” and the insights and theorems in this paper might enable future work in that direction.