Answers.Body: <p>There are two types of dropout, depending on whether a scaling correction is applied during:</p>
  <ul>
  <li><p><em>testing</em> - without dropout applied, to decrease logits by a factor <span class="math-container">$1-p$</span> to match the expected magnitude that layers were trained with.</p>
  </li>
  <li><p><em>training</em> - with dropout applied to increase logits by a factor <span class="math-container">$\frac{1}{1-p}$</span> to match expected magnitude without dropout.</p>
  </li>
  </ul>
  <p>(Both these assume <span class="math-container">$p$</span> is your probability that a node value is dropped out. Some will use <span class="math-container">$p$</span> as the probability that a node value is kept).</p>
  <p>The first version above was the original invention of dropout, but it has fallen out of favour nowadays, and the second one - sometimes called <a href="https://machinelearning.wtf/terms/inverted-dropout" rel="nofollow noreferrer">inverted dropout</a> - is more commonly used. The advantage of inverted dropout is that it is purely a training adjustment and the whole dropout layer can be removed for production use.</p>
  <p>If you do not apply the correct numerical adjustment, then networks with dropout layers will perform poorly.</p>
  <p>To answer your question title:</p>
  <blockquote>
  <p>Is it mandatory to multiply every activation of a layer by droupout factor during testing?</p>
  </blockquote>
  <p>Only if you are using the original formulation of dropout. Many neural network libraries use inverted dropout, so do not do this. Those libraries have related adjustment during training instead.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-30T10:05:16.660Z
Answers.Text: So, it is mandatory to do.
Answers.UserDisplayName: hanugm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-30T10:55:41.560Z
Answers.Text: @hanugm Yes, if by "mandatory" you mean "the algorithm doesn't work without it". However, you should check which version you are using. Inverted dropout does not apply any correction in testing phase, by design
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-30T08:13:40.530Z
Answers.LastActivityDate: 2022-05-30T10:58:22.497Z
Answers.LastEditDate: 2022-05-30T10:58:22.497Z
Answers.UserDisplayName: Neil Slater
Body: <p>Dropout is a regularization technique used in neural networks. It is useful in preventing overfitting by making a neural network as good as an ensemble system.</p>
  <p>In dropout, we switch off <span class="math-container">$p$</span> percent of neurons of the input or hidden layer during the training phase.  But, we do not switch off them during the testing phase.</p>
  <p>We generally multiply each activation by <span class="math-container">$p$</span>, the dropout ratio used during the training phase, in the testing phase. Is it mandatory to do so? Or is it okay to use the whole network as it is?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-05-30T01:18:32.607Z
Comments.Text: Isn't it 1-p? Also, it is mandatory to do so from what I know, except there are other ways to ensure the scale of the weights are the same (such as inverted dropout).
Comments.UserDisplayName: DKDK
Tags: <regularization><testing><dropout>
Title: Is it mandatory to multiply every activation of a layer by droupout factor during testing?
UserDisplayName: hanugm
ThreadText: # Is it mandatory to multiply every activation of a layer by droupout factor during testing?
  
  (Asked by: username_0 on 2022-05-29)
  
  Dropout is a regularization technique used in neural networks. It is useful in preventing overfitting by making a neural network as good as an ensemble system.
  
  In dropout, we switch off $p$ percent of neurons of the input or hidden layer during the training phase. But, we do not switch off them during the testing phase.
  
  We generally multiply each activation by $p$, the dropout ratio used during the training phase, in the testing phase. Is it mandatory to do so? Or is it okay to use the whole network as it is?
  
  **Tags:** regularization, testing, dropout
  
  > **Comments:**
  > **username_1**: Isn't it 1-p? Also, it is mandatory to do so from what I know, except there are other ways to ensure the scale of the weights are the same (such as inverted dropout).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-05-30. Score: 1
  
  There are two types of dropout, depending on whether a scaling correction is applied during:
  
  * *testing* - without dropout applied, to decrease logits by a factor $1-p$ to match the expected magnitude that layers were trained with.
  * *training* - with dropout applied to increase logits by a factor $\frac{1}{1-p}$ to match expected magnitude without dropout.
  
  (Both these assume $p$ is your probability that a node value is dropped out. Some will use $p$ as the probability that a node value is kept).
  
  The first version above was the original invention of dropout, but it has fallen out of favour nowadays, and the second one - sometimes called [inverted dropout](https://machinelearning.wtf/terms/inverted-dropout) - is more commonly used. The advantage of inverted dropout is that it is purely a training adjustment and the whole dropout layer can be removed for production use.
  
  If you do not apply the correct numerical adjustment, then networks with dropout layers will perform poorly.
  
  To answer your question title:
  
  > Is it mandatory to multiply every activation of a layer by droupout factor during testing?
  
  Only if you are using the original formulation of dropout. Many neural network libraries use inverted dropout, so do not do this. Those libraries have related adjustment during training instead.
  
  > **Comments:**
  > **username_0**: So, it is mandatory to do.  
  > **username_2**: @username_0 Yes, if by "mandatory" you mean "the algorithm doesn't work without it". However, you should check which version you are using. Inverted dropout does not apply any correction in testing phase, by design  
  
  ---
  
  