Answers.Body: <p>The idea of dropout is that, <strong>at training time</strong>, with a certain probability <span class="math-container">$p_i \in [0, 1]$</span>, the unit (or neuron) <span class="math-container">$i$</span> is dropped, <span class="math-container">$\forall i$</span>, that is, the output of unit <span class="math-container">$i$</span> is set to zero so that <span class="math-container">$i$</span> does not affect the other units it is connected to, both during the forward and backward (or back-propagation) passes (or steps). At every mini-batch, you randomly drop usually different units, so, across different mini-batches (and consequently epochs), you do <strong>not</strong> always or necessarily drop the same units. </p>
  
  <p>The title of the paper <a href="https://arxiv.org/pdf/1207.0580.pdf" rel="nofollow noreferrer">Improving neural networks <strong>by preventing co-adaptation of feature detectors</strong></a> emphasizes that dropout prevents the co-adaptation of the units (the feature detectors), so units attempt to detect certain features independently of other units, which reduces overfitting, that is, it improves the generalization ability of the neural network.</p>
  
  <p>At test time, no unit is usually dropped. However, there is an approximation of a <a href="http://proceedings.mlr.press/v31/damianou13a.pdf" rel="nofollow noreferrer">deep Gaussian process</a> and <a href="https://arxiv.org/abs/1505.05424" rel="nofollow noreferrer">Bayesian neural network</a> that is based on the application of dropout at training and test times. This is called <strong>Monte Carlo dropout</strong> or, in short, MC dropout, for reasons you can understand if you read the paper <a href="https://arxiv.org/pdf/1506.02142.pdf" rel="nofollow noreferrer">Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning</a>. </p>
  
  <p>There's also the possibility to drop the connections between the neurons, which is called <a href="http://proceedings.mlr.press/v28/wan13.html" rel="nofollow noreferrer">DropConnect</a>, rather than the neurons themselves. These two approaches are slightly different, even though DropConnect can be seen as a generalization of dropout. In DropConnect, you do not switch off completely the units, but only the contributions of certain units to the output of other units. In dropout, you completely switch off certain units.</p>
  
  <p>If you decided to <strong>deterministically</strong> (and manually) reduce the number of units before training, essentially, you would train a <strong>fixed</strong> smaller network, but this will not necessarily reduce overfitting or, more precisely, co-adaptation of the units. In dropout, you <strong>randomly</strong> select the units to drop, so, at every mini-batch (or epoch, depending on the implementation of dropout), you effectively train a <strong>random</strong> subset of the units of the original neural network and, because of this, it can be thought of as an ensemble of smaller neural networks.</p>
  
  <p>The two papers <a href="https://arxiv.org/pdf/1207.0580.pdf" rel="nofollow noreferrer">Improving neural networks by preventing co-adaptation of feature detectors</a> (2012) and <a href="http://jmlr.org/papers/volume15/srivastava14a.old/srivastava14a.pdf" rel="nofollow noreferrer">Dropout: A Simple Way to Prevent Neural Networks from Overfitting</a> (2014) have exactly the same authors, but the latter paper was published in the <a href="http://www.jmlr.org/" rel="nofollow noreferrer">Journal of Machine Learning Research</a>, while the former wasn't apparently published in any journal. In fact, <a href="http://jmlr.org/papers/volume15/srivastava14a.old/srivastava14a.pdf" rel="nofollow noreferrer">Dropout: A Simple Way to Prevent Neural Networks from Overfitting</a> does not even cite <a href="https://arxiv.org/pdf/1207.0580.pdf" rel="nofollow noreferrer">Improving neural networks by preventing co-adaptation of feature detectors</a>, but it cites the master's thesis <a href="http://www.cs.toronto.edu/~nitish/msc_thesis.pdf" rel="nofollow noreferrer">Improving Neural Networks with Dropout</a> (2013) by Nitish Srivastava, who is one of the authors of dropout.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-12T22:10:20.660Z
Answers.Text: (+1) But not that it's not necessarily true that *different* units are dropped at each training iteration. Vanilla dropout is essentially a Bernoulli(p) binary mask, so the same unit can be dropped during 2 successive iterations with probability $p^2$. Likewise, the entire dropout mask could, by chance, be the same between two iterations with some small probability.
Answers.UserDisplayName: Sycorax
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-12T22:12:57.203Z
Answers.Text: @SycoraxsaysReinstateMonica Thanks for the feedback. I've started this answer with "The idea of dropout", because I wanted more to give the intuition. However, you're right, the same units could be dropped during two successive training iterations. I've edited my answer to try accommodating the information in your comment.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-12T21:01:00.050Z
Answers.LastActivityDate: 2019-12-12T22:13:52.277Z
Answers.LastEditDate: 2019-12-12T22:13:52.277Z
Answers.UserDisplayName: nbro
Answers.Body: <p>Dropout only ignores a portion of units during a single training batch update.  Each training batch will use a different combination of units which gives them the best chance of that portion of them working together to generalize. Note the weights for each unit are kept and will be updated during the next batch in which that unit is selected. During inference, yes, all units are used (with a factor applied to activation...the same factor that defines the fraction used)...this becomes essentially an ensemble of all the different combinations of units that were used.</p>
  <p>Contrasted with fewer units, the fewer units approach will only learn what those fewer units can be optimized for.  Think of dropout as an ensemble of layers of fewer units.(with the exception that there are partial weight sharing between the layers)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-02T03:36:46.903Z
Answers.Text: I believe the comment about "each training batch" using a "different combination of units" is incorrect, at least as per the original paper on dropout. The dropout is applied per observation, not per mini batch. I believe this also means that weights of units that are dropped for only some of the observations in a mini batch are indeed updated during back propagation because they contribute to the loss function, which is an average across all observations in a mini batch. See: https://stats.stackexchange.com/questions/335690/understanding-dropout-method-one-mask-per-batch-or-more.
Answers.UserDisplayName: Fijoy Vadakkumpadan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-12T20:15:21.563Z
Answers.LastActivityDate: 2021-04-13T15:55:11.737Z
Answers.LastEditDate: 2021-04-13T15:55:11.737Z
Answers.UserDisplayName: user1269942
Body: <p>Why is dropout favored compared to reducing the number of units in hidden layers for the convolutional networks?</p>
  
  <p>If a large set of units leads to overfitting and dropping out "averages" the response units, why not just suppress units?</p>
  
  <p>I have read different questions and answers on the dropout topic including these interesting ones, <a href="https://ai.stackexchange.com/q/40/2444">What is the &quot;dropout&quot; technique?</a> and this other <a href="https://ai.stackexchange.com/q/9512/2444">Should I remove the units of a neural network or increase dropout?</a>, but did not get the proper answer to my question.</p>
  
  <p>By the way, it is weird that this publication <a href="http://jmlr.org/papers/volume15/srivastava14a.old/srivastava14a.pdf" rel="noreferrer">A Simple Way to Prevent Neural Networks from Overfitting (2014)</a>, Nitish Srivastava et al., is cited as being the first on the subject. I have just read one that is from 2012:
  <a href="https://arxiv.org/pdf/1207.0580.pdf" rel="noreferrer">Improving neural networks by preventing co-adaptation of feature detectors</a>.</p>
  
Tags: <neural-networks><machine-learning><deep-learning><dropout><regularization>
Title: Why is dropout favoured compared to reducing the number of units in hidden layers?
UserDisplayName: pascal sautot
ThreadText: # Why is dropout favoured compared to reducing the number of units in hidden layers?
  
  (Asked by: username_0 on 2019-12-11)
  
  Why is dropout favored compared to reducing the number of units in hidden layers for the convolutional networks?
  
  If a large set of units leads to overfitting and dropping out "averages" the response units, why not just suppress units?
  
  I have read different questions and answers on the dropout topic including these interesting ones, [What is the "dropout" technique?](https://ai.stackexchange.com/q/40/2444) and this other [Should I remove the units of a neural network or increase dropout?](https://ai.stackexchange.com/q/9512/2444), but did not get the proper answer to my question.
  
  By the way, it is weird that this publication [A Simple Way to Prevent Neural Networks from Overfitting (2014)](http://jmlr.org/papers/volume15/srivastava14a.old/srivastava14a.pdf), Nitish Srivastava et al., is cited as being the first on the subject. I have just read one that is from 2012:
  [Improving neural networks by preventing co-adaptation of feature detectors](https://arxiv.org/pdf/1207.0580.pdf).
  
  **Tags:** neural-networks, machine-learning, deep-learning, dropout, regularization
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2019-12-12. Score: 7 | Accepted answer
  
  Dropout only ignores a portion of units during a single training batch update. Each training batch will use a different combination of units which gives them the best chance of that portion of them working together to generalize. Note the weights for each unit are kept and will be updated during the next batch in which that unit is selected. During inference, yes, all units are used (with a factor applied to activation...the same factor that defines the fraction used)...this becomes essentially an ensemble of all the different combinations of units that were used.
  
  Contrasted with fewer units, the fewer units approach will only learn what those fewer units can be optimized for. Think of dropout as an ensemble of layers of fewer units.(with the exception that there are partial weight sharing between the layers)
  
  > **Comments:**
  > **username_4**: I believe the comment about "each training batch" using a "different combination of units" is incorrect, at least as per the original paper on dropout. The dropout is applied per observation, not per mini batch. I believe this also means that weights of units that are dropped for only some of the observations in a mini batch are indeed updated during back propagation because they contribute to the loss function, which is an average across all observations in a mini batch. See: https://stats.stackexchange.com/questions/335690/understanding-dropout-method-one-mask-per-batch-or-more.  
  
  ---
  
  ### Answer by username_1 on 2019-12-12. Score: 2
  
  The idea of dropout is that, **at training time**, with a certain probability $p\_i \in [0, 1]$, the unit (or neuron) $i$ is dropped, $\forall i$, that is, the output of unit $i$ is set to zero so that $i$ does not affect the other units it is connected to, both during the forward and backward (or back-propagation) passes (or steps). At every mini-batch, you randomly drop usually different units, so, across different mini-batches (and consequently epochs), you do **not** always or necessarily drop the same units.
  
  The title of the paper [Improving neural networks **by preventing co-adaptation of feature detectors**](https://arxiv.org/pdf/1207.0580.pdf) emphasizes that dropout prevents the co-adaptation of the units (the feature detectors), so units attempt to detect certain features independently of other units, which reduces overfitting, that is, it improves the generalization ability of the neural network.
  
  At test time, no unit is usually dropped. However, there is an approximation of a [deep Gaussian process](http://proceedings.mlr.press/v31/damianou13a.pdf) and [Bayesian neural network](https://arxiv.org/abs/1505.05424) that is based on the application of dropout at training and test times. This is called **Monte Carlo dropout** or, in short, MC dropout, for reasons you can understand if you read the paper [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://arxiv.org/pdf/1506.02142.pdf).
  
  There's also the possibility to drop the connections between the neurons, which is called [DropConnect](http://proceedings.mlr.press/v28/wan13.html), rather than the neurons themselves. These two approaches are slightly different, even though DropConnect can be seen as a generalization of dropout. In DropConnect, you do not switch off completely the units, but only the contributions of certain units to the output of other units. In dropout, you completely switch off certain units.
  
  If you decided to **deterministically** (and manually) reduce the number of units before training, essentially, you would train a **fixed** smaller network, but this will not necessarily reduce overfitting or, more precisely, co-adaptation of the units. In dropout, you **randomly** select the units to drop, so, at every mini-batch (or epoch, depending on the implementation of dropout), you effectively train a **random** subset of the units of the original neural network and, because of this, it can be thought of as an ensemble of smaller neural networks.
  
  The two papers [Improving neural networks by preventing co-adaptation of feature detectors](https://arxiv.org/pdf/1207.0580.pdf) (2012) and [Dropout: A Simple Way to Prevent Neural Networks from Overfitting](http://jmlr.org/papers/volume15/srivastava14a.old/srivastava14a.pdf) (2014) have exactly the same authors, but the latter paper was published in the [Journal of Machine Learning Research](http://www.jmlr.org/), while the former wasn't apparently published in any journal. In fact, [Dropout: A Simple Way to Prevent Neural Networks from Overfitting](http://jmlr.org/papers/volume15/srivastava14a.old/srivastava14a.pdf) does not even cite [Improving neural networks by preventing co-adaptation of feature detectors](https://arxiv.org/pdf/1207.0580.pdf), but it cites the master's thesis [Improving Neural Networks with Dropout](http://www.cs.toronto.edu/~nitish/msc_thesis.pdf) (2013) by Nitish Srivastava, who is one of the authors of dropout.
  
  > **Comments:**
  > **username_2**: (+1) But not that it's not necessarily true that *different* units are dropped at each training iteration. Vanilla dropout is essentially a Bernoulli(p) binary mask, so the same unit can be dropped during 2 successive iterations with probability $p^2$. Likewise, the entire dropout mask could, by chance, be the same between two iterations with some small probability.  
  > **username_1**: @SycoraxsaysReinstateMonica Thanks for the feedback. I've started this answer with "The idea of dropout", because I wanted more to give the intuition. However, you're right, the same units could be dropped during two successive training iterations. I've edited my answer to try accommodating the information in your comment.  
  
  ---
  
  