Answers.Body: <p>The original paper<sup>1</sup> that proposed neural network dropout is titled: <a href="http://www.jmlr.org/papers/volume15/srivastava14a.old/source/srivastava14a.pdf" rel="nofollow noreferrer">Dropout: A simple way to prevent neural networks from overfitting</a>. That tittle pretty much explains in one sentence what Dropout does. Dropout works by randomly selecting and removing neurons in a neural network during the training phase. Note that dropout is not applied during testing and that the resulting network doesn't dropout as part of predicting. </p>
  
  <p>This random removal/dropout of neurons prevents excessive co-adaption of the neurons and in so doing, reduce the likelihood of the network <a href="https://en.wikipedia.org/wiki/Overfitting" rel="nofollow noreferrer">overfiting</a>. </p>
  
  <p>The random removal of neurons during training also means that at any point in time, only a portion of the original network is trained. This has the effect that you end up sort of training multiple sub-networks, for example:</p>
  
  <p><img src="https://i.sstatic.net/vMidK.jpg" alt="droup as an ensembler"></p>
  
  <p>It is from this repeated training of sub-networks as opposed to the entire network where the notion of neural network dropout being a sort of ensemble technique comes in. I.e the training of the sub-networks is similar to training numerous, relatively weak algorithms/models and combining them to form one algorithm that is more powerful than the individual parts.  </p>
  
  <p><strong><em>References:</em></strong> </p>
  
  <p><sub><sup>1</sup>: Srivastava, Nitish, et al. "Dropout: A simple way to prevent neural networks from overfitting." The Journal of Machine Learning Research 15.1 (2014): 1929-1958.</sub></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-05T16:26:08.093Z
Answers.Text: Can you elaborate on what co-adaptation of neurons means?
Answers.UserDisplayName: Antonios Sarikas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-05T22:36:39.773Z
Answers.Text: "Dropout works by randomly selecting and removing neurons in a neural network". Really, only the fully-connected part of a neural network.
Answers.UserDisplayName: Monica Heddneck
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-07T07:48:41.353Z
Answers.LastActivityDate: 2018-04-12T02:30:04.720Z
Answers.LastEditDate: 2018-04-12T02:30:04.720Z
Answers.UserDisplayName: Tshilidzi Mudau
Answers.Body: <p>There are some great answers here. The simplest explanation I can give for dropout is that it randomly excludes some neurons and their connections from the network, while training, to stop neurons from "co-adapting" too much. It has the effect of making each neuron apply more generally and is excellent for stopping overfitting for large neural networks.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-07T15:23:39.500Z
Answers.LastActivityDate: 2017-06-07T15:23:39.500Z
Answers.UserDisplayName: thegreenpizza
Answers.Body: <p>I'll try to answer your questions using Geoffrey Hinton's ideas in dropout paper and his Coursera class.</p>
  <p><strong>What purpose does the &quot;dropout&quot; method serve?</strong></p>
  <blockquote>
  <p>Deep neural nets with a large number of parameters are very powerful machine learning systems. However, overfitting is a serious problem in such networks. Large networks are also slow to use, making it difficult to deal with overfitting by combining the predictions of many different large neural nets at test time. Dropout is a technique for addressing this problem.</p>
  </blockquote>
  <p>So, dropout is a regularization technique that addresses the problem of overfitting.</p>
  <p><strong>How does it improve the overall performance?</strong> If by &quot;performance&quot; you mean &quot;generalization&quot;, then it may improve generalization by avoiding over-fitting, as I said.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2017-01-25T00:59:49.040Z
Answers.LastActivityDate: 2020-10-01T20:16:56.117Z
Answers.LastEditDate: 2020-10-01T20:16:56.117Z
Answers.UserDisplayName: Iman Mirzadeh
Answers.Body: <p>Dropout means that every individual data point is only used to fit a random subset of the neurons. This is done to make the neural network more like an ensemble model.</p>
  
  <p>That is, just as a random forest is averaging together the results of many individual decision trees, you can see a neural network trained using dropout as averaging together the results of many individual neural networks (with 'results' understood to mean activations at every layer, rather than just the output layer).</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-02T16:12:08.767Z
Answers.LastActivityDate: 2018-01-17T00:30:27.303Z
Answers.LastEditDate: 2018-01-17T00:30:27.303Z
Answers.UserDisplayName: Matthew Gray
Body: <p>What purpose does the "dropout" method serve and how does it improve the overall performance of the neural network?</p>
  
Tags: <deep-neural-networks><overfitting><performance>
Title: What is the "dropout" technique?
UserDisplayName: kenorb
ThreadText: # What is the "dropout" technique?
  
  (Asked by: username_0 on 2016-08-02)
  
  What purpose does the "dropout" method serve and how does it improve the overall performance of the neural network?
  
  **Tags:** deep-neural-networks, overfitting, performance
  
  ---
  
  ## 4 answers
  
  ### Answer by username_6 on 2016-08-02. Score: 8 | Accepted answer
  
  Dropout means that every individual data point is only used to fit a random subset of the neurons. This is done to make the neural network more like an ensemble model.
  
  That is, just as a random forest is averaging together the results of many individual decision trees, you can see a neural network trained using dropout as averaging together the results of many individual neural networks (with 'results' understood to mean activations at every layer, rather than just the output layer).
  
  
  ---
  
  ### Answer by username_1 on 2017-06-07. Score: 6
  
  The original paper1 that proposed neural network dropout is titled: [Dropout: A simple way to prevent neural networks from overfitting](http://www.jmlr.org/papers/volume15/srivastava14a.old/source/srivastava14a.pdf). That tittle pretty much explains in one sentence what Dropout does. Dropout works by randomly selecting and removing neurons in a neural network during the training phase. Note that dropout is not applied during testing and that the resulting network doesn't dropout as part of predicting.
  
  This random removal/dropout of neurons prevents excessive co-adaption of the neurons and in so doing, reduce the likelihood of the network [overfiting](https://en.wikipedia.org/wiki/Overfitting).
  
  The random removal of neurons during training also means that at any point in time, only a portion of the original network is trained. This has the effect that you end up sort of training multiple sub-networks, for example:
  
  ![droup as an ensembler](https://i.sstatic.net/vMidK.jpg)
  
  It is from this repeated training of sub-networks as opposed to the entire network where the notion of neural network dropout being a sort of ensemble technique comes in. I.e the training of the sub-networks is similar to training numerous, relatively weak algorithms/models and combining them to form one algorithm that is more powerful than the individual parts.
  
  ***References:***
  
  1: Srivastava, Nitish, et al. "Dropout: A simple way to prevent neural networks from overfitting." The Journal of Machine Learning Research 15.1 (2014): 1929-1958.
  
  > **Comments:**
  > **username_3**: "Dropout works by randomly selecting and removing neurons in a neural network". Really, only the fully-connected part of a neural network.  
  > **username_2**: Can you elaborate on what co-adaptation of neurons means?  
  
  ---
  
  ### Answer by username_4 on 2017-06-07. Score: 3
  
  There are some great answers here. The simplest explanation I can give for dropout is that it randomly excludes some neurons and their connections from the network, while training, to stop neurons from "co-adapting" too much. It has the effect of making each neuron apply more generally and is excellent for stopping overfitting for large neural networks.
  
  
  ---
  
  ### Answer by username_5 on 2017-01-25. Score: 2
  
  I'll try to answer your questions using Geoffrey Hinton's ideas in dropout paper and his Coursera class.
  
  **What purpose does the "dropout" method serve?**
  
  > Deep neural nets with a large number of parameters are very powerful machine learning systems. However, overfitting is a serious problem in such networks. Large networks are also slow to use, making it difficult to deal with overfitting by combining the predictions of many different large neural nets at test time. Dropout is a technique for addressing this problem.
  
  So, dropout is a regularization technique that addresses the problem of overfitting.
  
  **How does it improve the overall performance?** If by "performance" you mean "generalization", then it may improve generalization by avoiding over-fitting, as I said.
  
  
  ---
  
  