Answers.Body: <p>Backpropagation on a network with dropout works just as it does normally, it calculates the gradients and updates the weights.</p>
  <p><em>Longer explanation</em></p>
  <p>Dropout is a regularization technique which drops nodes in the forward pass.</p>
  <p>In the backward pass, the 'influence' of every weight on the end result is calculated  (the gradient). If a node was dropped by the Dropout layer, then its influence of the outgoing weights is therefore also 0 (as 0 * weight = 0). In short, backpropagation works just as it always does.</p>
  <p>So how does it learn, if those nodes are never updated? What you need to understand is that in every forward pass, the nodes which are dropped are random. And hence, in every forward pass, different nodes are dropped. As a result, in every backward pass, different nodes are updated.</p>
  <p>Because nodes are dropped, the network cannot rely solely on specific nodes to make predictions, and hence the network (should) generalize better. But I think you already understood that part.</p>
  <p><a href="https://deepnotes.io/dropout#:%7E:text=Dropout%20works%20by%20randomly%20and,the%20result%20through%20the%20same." rel="nofollow noreferrer">For an even more elaborate explanation.</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-13T14:15:57.400Z
Answers.Text: Thank you. Does that mean, as long as your new weights are stored somewhere, the back propagation part of the algorithm will work automatically? Or do I have to do something else when implementing this part of the algorithm?
Answers.UserDisplayName: Connor
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-13T14:29:07.877Z
Answers.Text: I'm not sure whether your implementation works or does not work without more specifications. However, if you have implementation/code-specific questions, those should be placed on StackOverflow. So I'd suggest you post a minimal code example there if you need assistance with coding.
Answers.UserDisplayName: Robin van Hoorn
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-13T14:46:50.090Z
Answers.Text: Okay, I'm not really worried about coding it, that part is easy, I'm just wondering if I get the backpropagation part of the algorithm for free. Could you extend your answer to add something about that? I'll happily accept afterwards.
Answers.UserDisplayName: Connor
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-13T14:51:31.267Z
Answers.Text: Could you specify your question then? Do you mean what you have to do 'extra' to the backpropagation algorithm if you have a dropout layer? The answer to that question is simply 'nothing' :)
Answers.UserDisplayName: Robin van Hoorn
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-13T15:02:09.510Z
Answers.Text: Basically, that's all I want, but with slightly more academic justification. Perhaps I'm being particular, but I don't want to implement a bad algorithm because I'm unsure of what I'm doing!
Answers.UserDisplayName: Connor
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-13T15:07:53.020Z
Answers.Text: No worries. Did a quick search, but am unsure of what academic justification you search. Something like https://www.cs.toronto.edu/~rsalakhu/papers/srivastava14a.pdf describes backpropagation with dropout.
Answers.UserDisplayName: Robin van Hoorn
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-13T15:15:17.430Z
Answers.Text: It states that "Forward and backpropagation [...] are done only on this thinned network" referring to a network where the nodes are dropped out at random.
Answers.UserDisplayName: Robin van Hoorn
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-25T19:55:19.130Z
Answers.Text: @RobinvanHoorn Might a naive question, but how a network differentiates between a neuron with output 0 and a dropped-out neuron (this neuron might output non-zero value but due to dropout it outputs 0). Imaging you are a neuron and you are receiving the output from the previous layer. Whether it was dropped out to 0 or 0 (without dropout) it doesn't make any difference at all.
Answers.UserDisplayName: Antonios Sarikas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-26T07:14:31.117Z
Answers.Text: You are correct. This is a common situation called often referred to with terms such as 'dead weights' or 'dead neurons'. The model does not make distinctions between them, and effectively treats them the same
Answers.UserDisplayName: Robin van Hoorn
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-30T20:31:35.840Z
Answers.Text: @RobinvanHoorn Something I would like to add is that there is a difference during the backward pass since gradient can still backprop through the non-dropped out neuron. In general, the neuron will take as input a vector $\mathbf{x}$ and if any element $x_i \neq 0$, then gradient can still backpropagated. (Except the special case where the neuron takes as input a single value $x$ which happens to be equal to 0).
Answers.UserDisplayName: Antonios Sarikas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-13T14:00:29.660Z
Answers.LastActivityDate: 2022-12-13T14:52:41.283Z
Answers.LastEditDate: 2022-12-13T14:52:41.283Z
Answers.UserDisplayName: Robin van Hoorn
Body: <p>I've searched for an answer to this, and read several scientific articles on the subject, but I can't find a practical explanation of how Dropout actually drops nodes in an algorithm.</p>
  <p>I've read that Dropout zeros out the activation function for particular nodes, which makes sense in the forward pass. But how does this work for the backward pass?</p>
  
Tags: <neural-networks><machine-learning><regularization><dropout>
Title: How does dropout work during backpropagation?
UserDisplayName: Connor
ThreadText: # How does dropout work during backpropagation?
  
  (Asked by: username_0 on 2022-12-13)
  
  I've searched for an answer to this, and read several scientific articles on the subject, but I can't find a practical explanation of how Dropout actually drops nodes in an algorithm.
  
  I've read that Dropout zeros out the activation function for particular nodes, which makes sense in the forward pass. But how does this work for the backward pass?
  
  **Tags:** neural-networks, machine-learning, regularization, dropout
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-12-13. Score: 4 | Accepted answer
  
  Backpropagation on a network with dropout works just as it does normally, it calculates the gradients and updates the weights.
  
  *Longer explanation*
  
  Dropout is a regularization technique which drops nodes in the forward pass.
  
  In the backward pass, the 'influence' of every weight on the end result is calculated (the gradient). If a node was dropped by the Dropout layer, then its influence of the outgoing weights is therefore also 0 (as 0 \* weight = 0). In short, backpropagation works just as it always does.
  
  So how does it learn, if those nodes are never updated? What you need to understand is that in every forward pass, the nodes which are dropped are random. And hence, in every forward pass, different nodes are dropped. As a result, in every backward pass, different nodes are updated.
  
  Because nodes are dropped, the network cannot rely solely on specific nodes to make predictions, and hence the network (should) generalize better. But I think you already understood that part.
  
  [For an even more elaborate explanation.](https://deepnotes.io/dropout#:%7E:text=Dropout%20works%20by%20randomly%20and,the%20result%20through%20the%20same.)
  
  > **Comments:**
  > **username_0**: Thank you. Does that mean, as long as your new weights are stored somewhere, the back propagation part of the algorithm will work automatically? Or do I have to do something else when implementing this part of the algorithm?  
  > **username_1**: I'm not sure whether your implementation works or does not work without more specifications. However, if you have implementation/code-specific questions, those should be placed on StackOverflow. So I'd suggest you post a minimal code example there if you need assistance with coding.  
  > **username_0**: Okay, I'm not really worried about coding it, that part is easy, I'm just wondering if I get the backpropagation part of the algorithm for free. Could you extend your answer to add something about that? I'll happily accept afterwards.  
  > **username_1**: Could you specify your question then? Do you mean what you have to do 'extra' to the backpropagation algorithm if you have a dropout layer? The answer to that question is simply 'nothing' :)  
  > **username_0**: Basically, that's all I want, but with slightly more academic justification. Perhaps I'm being particular, but I don't want to implement a bad algorithm because I'm unsure of what I'm doing!  
  > **username_1**: No worries. Did a quick search, but am unsure of what academic justification you search. Something like https://www.cs.toronto.edu/~rsalakhu/papers/srivastava14a.pdf describes backpropagation with dropout.  
  > **username_1**: It states that "Forward and backpropagation [...] are done only on this thinned network" referring to a network where the nodes are dropped out at random.  
  > **username_2**: @RobinvanHoorn Might a naive question, but how a network differentiates between a neuron with output 0 and a dropped-out neuron (this neuron might output non-zero value but due to dropout it outputs 0). Imaging you are a neuron and you are receiving the output from the previous layer. Whether it was dropped out to 0 or 0 (without dropout) it doesn't make any difference at all.  
  > **username_1**: You are correct. This is a common situation called often referred to with terms such as 'dead weights' or 'dead neurons'. The model does not make distinctions between them, and effectively treats them the same  
  > **username_2**: @RobinvanHoorn Something I would like to add is that there is a difference during the backward pass since gradient can still backprop through the non-dropped out neuron. In general, the neuron will take as input a vector $\mathbf{x}$ and if any element $x_i \neq 0$, then gradient can still backpropagated. (Except the special case where the neuron takes as input a single value $x$ which happens to be equal to 0).  
  
  ---
  
  