Answers.Body: <p>Dropout is a technique that helps to avoid overfitting during training. That is, dropout is usually used for training.</p>
  <blockquote>
  <p>units may change in a way that they fix up the mistakes of the other
  units. This may lead to complex co-adaptations. This, in turn, leads to
  overfitting because these co-adaptations do not generalize to unseen
  data.</p>
  </blockquote>
  <p>If you want to evaluate your model, you should turn off all dropout layers. For example, PyTorch's <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.Module.eval" rel="nofollow noreferrer"><code>model.eval()</code></a> does this work.</p>
  <p>Note that in some cases dropout can be used for inference, e.g. to add some stochasticity to the output.</p>
  <p>More about dropout:</p>
  <ul>
  <li><a href="https://arxiv.org/abs/1207.0580" rel="nofollow noreferrer">Improving neural networks by preventing co-adaptation of feature detectors</a></li>
  <li><a href="http://jmlr.org/papers/v15/srivastava14a.html" rel="nofollow noreferrer">Dropout: A Simple Way to Prevent Neural Networks from Overfitting</a></li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-03T16:18:40.373Z
Answers.Text: I did not know PyTorch automatically deactivated drop out layers
Answers.UserDisplayName: Tobi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-03T16:16:01.083Z
Answers.LastActivityDate: 2022-05-13T09:30:40.297Z
Answers.LastEditDate: 2022-05-13T09:30:40.297Z
Answers.UserDisplayName: Aray Karjauv
Answers.Body: <p><a href="http://jmlr.org/papers/volume15/srivastava14a/srivastava14a.pdf" rel="nofollow noreferrer">Dropout</a> is usually disabled at test (or evaluation) time. For example, in Keras, dropout is disabled at evaluation time by default, although you can enable it, if you need to (see below). The purpose of dropout is to decorrelate the units (or feature detectors) so that they learn more robust representations of the data (i.e. a form of regularisation). </p>
  
  <p>However, there's also <a href="https://arxiv.org/pdf/1506.02142.pdf" rel="nofollow noreferrer">Monte Carlo (MC) dropout</a>, i.e., you train the network with dropout and you also use dropout at test time in order to get stochastic outputs (i.e. you will get different outputs, for different forward passes, given the same inputs). <a href="https://arxiv.org/pdf/1506.02142.pdf" rel="nofollow noreferrer">MC dropout is an approximation of Bayesian inference in deep Gaussian processes</a>, which means that MC dropout is roughly equivalent to a <a href="https://arxiv.org/pdf/1505.05424.pdf" rel="nofollow noreferrer">Bayesian neural network</a>.</p>
  
  <blockquote>
    <p>Does the performance of a model increase if dropout is disabled at evaluation time?</p>
  </blockquote>
  
  <p>Yes, possibly. However, MC dropout provides an uncertainty measure, which can be useful in certain scenarios (e.g. medical scenarios), where a point estimate (i.e. a single prediction or classification) is definitely not appropriate, but you also need a measure of the uncertainty or confidence of the predictions.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-03T17:43:42.483Z
Answers.LastActivityDate: 2020-03-03T17:55:35.677Z
Answers.LastEditDate: 2020-03-03T17:55:35.677Z
Answers.UserDisplayName: nbro
Body: <p>I know dropout layers are used in neural networks during training to provide a form of regularisation in an attempt to mitigate over-fitting.</p>
  
  <p>Would you not get an increased fitness if you disabled the dropout layers during evaluation of a network?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-03T17:23:06.297Z
Comments.Text: Keras turns off dropout when computing validation accuracy during training. Also when making predictions
Comments.UserDisplayName: Gerry P
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-03T17:49:29.010Z
Comments.Text: Follow-up question: https://ai.stackexchange.com/questions/18397/why-is-my-validation-test-accuracy-higher-than-my-training-accuracy
Comments.UserDisplayName: Tobi
Tags: <neural-networks><machine-learning><dropout>
Title: Does the performance of a model increase if dropout is disabled at evaluation time?
UserDisplayName: Tobi
ThreadText: # Does the performance of a model increase if dropout is disabled at evaluation time?
  
  (Asked by: username_0 on 2020-03-03)
  
  I know dropout layers are used in neural networks during training to provide a form of regularisation in an attempt to mitigate over-fitting.
  
  Would you not get an increased fitness if you disabled the dropout layers during evaluation of a network?
  
  **Tags:** neural-networks, machine-learning, dropout
  > **Comments:**
  > **username_1**: Keras turns off dropout when computing validation accuracy during training. Also when making predictions  
  > **username_0**: Follow-up question: https://ai.stackexchange.com/questions/18397/why-is-my-validation-test-accuracy-higher-than-my-training-accuracy  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2020-03-03. Score: 3
  
  Dropout is a technique that helps to avoid overfitting during training. That is, dropout is usually used for training.
  
  > units may change in a way that they fix up the mistakes of the other
  > units. This may lead to complex co-adaptations. This, in turn, leads to
  > overfitting because these co-adaptations do not generalize to unseen
  > data.
  
  If you want to evaluate your model, you should turn off all dropout layers. For example, PyTorch's [`model.eval()`](https://pytorch.org/docs/stable/nn.html#torch.nn.Module.eval) does this work.
  
  Note that in some cases dropout can be used for inference, e.g. to add some stochasticity to the output.
  
  More about dropout:
  
  * [Improving neural networks by preventing co-adaptation of feature detectors](https://arxiv.org/abs/1207.0580)
  * [Dropout: A Simple Way to Prevent Neural Networks from Overfitting](http://jmlr.org/papers/v15/srivastava14a.html)
  
  > **Comments:**
  > **username_0**: I did not know PyTorch automatically deactivated drop out layers  
  
  ---
  
  ### Answer by username_3 on 2020-03-03. Score: 2
  
  [Dropout](http://jmlr.org/papers/volume15/srivastava14a/srivastava14a.pdf) is usually disabled at test (or evaluation) time. For example, in Keras, dropout is disabled at evaluation time by default, although you can enable it, if you need to (see below). The purpose of dropout is to decorrelate the units (or feature detectors) so that they learn more robust representations of the data (i.e. a form of regularisation).
  
  However, there's also [Monte Carlo (MC) dropout](https://arxiv.org/pdf/1506.02142.pdf), i.e., you train the network with dropout and you also use dropout at test time in order to get stochastic outputs (i.e. you will get different outputs, for different forward passes, given the same inputs). [MC dropout is an approximation of Bayesian inference in deep Gaussian processes](https://arxiv.org/pdf/1506.02142.pdf), which means that MC dropout is roughly equivalent to a [Bayesian neural network](https://arxiv.org/pdf/1505.05424.pdf).
  
  > Does the performance of a model increase if dropout is disabled at evaluation time?
  
  Yes, possibly. However, MC dropout provides an uncertainty measure, which can be useful in certain scenarios (e.g. medical scenarios), where a point estimate (i.e. a single prediction or classification) is definitely not appropriate, but you also need a measure of the uncertainty or confidence of the predictions.
  
  
  ---
  
  