Answers.Body: <p>In the images below (figure 9 from section 7.3) <span class="math-container">$n$</span> refers to a unit in the network and <span class="math-container">$p$</span> refers to the probability of retaining a unit in the network. The combination of both, <span class="math-container">$pn$</span> thus refers to the &quot;effective&quot; number of units. The left image thus shows how training and test error change when varying the probability of retaining a unit (<span class="math-container">$p$</span>) whereby the number of units (<span class="math-container">$n$</span>) is fixed. The right image shows how the errors change when the &quot;effective&quot; number of units (<span class="math-container">$pn$</span>) is fixed, and the amount of units (<span class="math-container">$n$</span>) is adopted with an increasing <span class="math-container">$p$</span>.</p>
  <p>The test error is U-shaped with a sweet spot around 0.5 or 0.6 in both cases. In the left image where the number of units (<span class="math-container">$n$</span>) is fixed, this likely occurs since the ensemble-like properties of dropout networks do not emerge with too little dropout (<span class="math-container">$p$</span> below 0.3) or too much dropout (<span class="math-container">$p$</span> above 0.8). Those results could however be purely due to a change in capacity. The right image keeps the &quot;capacity&quot; constant and only looks at the &quot;pure dropout effect&quot; since the capacity (namely the number of units <span class="math-container">$pn$</span>) is fixed. This rules out a decrease in capacity as the driving factor. The interaction of the decrease in capacity due to an increase in dropout is hinted at with the following sentence:</p>
  <blockquote>
  <p>we notice that the magnitude of errors for small values of p has reduced by a lot compared to Figure 9a</p>
  </blockquote>
  <p><a href="https://i.sstatic.net/AIrG7.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/AIrG7.png" alt="enter image description here" /></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-13T22:28:58.407Z
Answers.LastActivityDate: 2023-02-13T22:28:58.407Z
Answers.UserDisplayName: Mariusmarten
Body: <p>In the <a href="https://jmlr.org/papers/volume15/srivastava14a/srivastava14a.pdf" rel="nofollow noreferrer">original paper on dropout</a>, in section 7.3.2, we see that while keeping <span class="math-container">$pn$</span> constant, we get a (test) error increase by decreasing retainment below 0.6.  Why would that happen? If <span class="math-container">$pn$</span> is constant, the capacity of the network should be constant, correct?</p>
  
Tags: <deep-learning><training><papers><regularization><dropout>
Title: In the Dropout paper, why would increasing the dropout increase the error rate if the capacity is constant?
UserDisplayName: Apples14
ThreadText: # In the Dropout paper, why would increasing the dropout increase the error rate if the capacity is constant?
  
  (Asked by: username_0 on 2023-02-09)
  
  In the [original paper on dropout](https://jmlr.org/papers/volume15/srivastava14a/srivastava14a.pdf), in section 7.3.2, we see that while keeping $pn$ constant, we get a (test) error increase by decreasing retainment below 0.6. Why would that happen? If $pn$ is constant, the capacity of the network should be constant, correct?
  
  **Tags:** deep-learning, training, papers, regularization, dropout
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-02-13. Score: 1
  
  In the images below (figure 9 from section 7.3) $n$ refers to a unit in the network and $p$ refers to the probability of retaining a unit in the network. The combination of both, $pn$ thus refers to the "effective" number of units. The left image thus shows how training and test error change when varying the probability of retaining a unit ($p$) whereby the number of units ($n$) is fixed. The right image shows how the errors change when the "effective" number of units ($pn$) is fixed, and the amount of units ($n$) is adopted with an increasing $p$.
  
  The test error is U-shaped with a sweet spot around 0.5 or 0.6 in both cases. In the left image where the number of units ($n$) is fixed, this likely occurs since the ensemble-like properties of dropout networks do not emerge with too little dropout ($p$ below 0.3) or too much dropout ($p$ above 0.8). Those results could however be purely due to a change in capacity. The right image keeps the "capacity" constant and only looks at the "pure dropout effect" since the capacity (namely the number of units $pn$) is fixed. This rules out a decrease in capacity as the driving factor. The interaction of the decrease in capacity due to an increase in dropout is hinted at with the following sentence:
  
  > we notice that the magnitude of errors for small values of p has reduced by a lot compared to Figure 9a
  
  [![enter image description here](https://i.sstatic.net/AIrG7.png)](https://i.sstatic.net/AIrG7.png)
  
  
  ---
  
  