Answers.Body: <p>The values need to be stored during each batch only, as the forward values are needed to calculate the gradients.</p>
  <p>A relevant equation from back propagation is how to derive <span class="math-container">$\nabla_{z_j^{(k)}}J$</span>, or the gradient of error function <span class="math-container">$J$</span> with respect to an individual pre-non-linearity (or logit) value <span class="math-container">$z$</span> for neuron <span class="math-container">$j$</span> in layer <span class="math-container">$k$</span>. In the following, <span class="math-container">$f(z)$</span> is the non-linearity, and <span class="math-container">$a_j$</span> is the output after the non-linearity, and you know <span class="math-container">$\nabla a_j^{(k)}$</span> (which you would derive from next layer, or if <span class="math-container">$k$</span> is the output layer, from calculating the gradient due to the loss function):</p>
  <p><span class="math-container">$$\nabla_{z_j^{(k)}}J = \frac{\partial E}{\partial z_j^{(k)}} = \frac{\partial E}{\partial a_j^{(k)}} \frac{\partial a_j^{(k)}}{\partial z_j^{(k)}} = \nabla a_j^{(k)} f'(z_j^{(k)})$$</span></p>
  <p>The value <span class="math-container">$z_j^{(k)}$</span> needs to be stored here to evaluate the gradient <span class="math-container">$\nabla_{z_j^{(k)}}J$</span>. The value of that gradient is then used in further equations for back-propagation in order to calculate the gradients of the weights, which are usually the parameters you want to update.</p>
  <p>The storage required is the number of nodes that feed forward, i.e. the total number of neurons in the network (technically also the input values for calculating the first set of gradients, but you will already be storing those, probably for longer), <em>times</em> the size of the batch.</p>
  <p>Once each batch or mini-batch has had gradients back-propagated, there is no further use for the node/neuron outputs, and the memory allocation can be re-used for the next batch.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-23T22:20:34.687Z
Answers.Text: To make sure I understood correctly: we need to store the forward values because 1) we need it to calculate at least the first gradient going back, and 2) for hidden units (for example, sigmoid) the gradient also contains that value in its expression? If yes, then can you direct me towards some proof intuitive or otherwise of the generalizability of the second point above?
Answers.UserDisplayName: Kunj Mehta
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-23T22:40:54.067Z
Answers.Text: @KunjMehta I added the equation from backprop that uses the value being stored. There are other backprop equations that use the result from that point on.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-23T22:15:20.097Z
Answers.LastActivityDate: 2022-03-23T22:39:51.813Z
Answers.LastEditDate: 2022-03-23T22:39:51.813Z
Answers.UserDisplayName: Neil Slater
Body: <p>I have been reading the Deep Learning book by Ian Goodfellow and it mentions in Section 6.5.7 that</p>
  <blockquote>
  <p>The main memory cost of the algorithm is that we need to store the input to the nonlinearity of the hidden layer.</p>
  </blockquote>
  <p>I understand that back-propagation stores the gradients in a similar fashion to dynamic programming so not to recompute them. But I am confused as to why it stores the input as well?</p>
  
Tags: <neural-networks><deep-learning><deep-neural-networks>
Title: Why does backprop algorithm store the inputs to the non-linearity of the hidden layers?
UserDisplayName: Kunj Mehta
ThreadText: # Why does backprop algorithm store the inputs to the non-linearity of the hidden layers?
  
  (Asked by: username_0 on 2022-03-23)
  
  I have been reading the Deep Learning book by Ian Goodfellow and it mentions in Section 6.5.7 that
  
  > The main memory cost of the algorithm is that we need to store the input to the nonlinearity of the hidden layer.
  
  I understand that back-propagation stores the gradients in a similar fashion to dynamic programming so not to recompute them. But I am confused as to why it stores the input as well?
  
  **Tags:** neural-networks, deep-learning, deep-neural-networks
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-03-23. Score: 1 | Accepted answer
  
  The values need to be stored during each batch only, as the forward values are needed to calculate the gradients.
  
  A relevant equation from back propagation is how to derive $\nabla\_{z\_j^{(k)}}J$, or the gradient of error function $J$ with respect to an individual pre-non-linearity (or logit) value $z$ for neuron $j$ in layer $k$. In the following, $f(z)$ is the non-linearity, and $a\_j$ is the output after the non-linearity, and you know $\nabla a\_j^{(k)}$ (which you would derive from next layer, or if $k$ is the output layer, from calculating the gradient due to the loss function):
  
  $$\nabla\_{z\_j^{(k)}}J = \frac{\partial E}{\partial z\_j^{(k)}} = \frac{\partial E}{\partial a\_j^{(k)}} \frac{\partial a\_j^{(k)}}{\partial z\_j^{(k)}} = \nabla a\_j^{(k)} f'(z\_j^{(k)})$$
  
  The value $z\_j^{(k)}$ needs to be stored here to evaluate the gradient $\nabla\_{z\_j^{(k)}}J$. The value of that gradient is then used in further equations for back-propagation in order to calculate the gradients of the weights, which are usually the parameters you want to update.
  
  The storage required is the number of nodes that feed forward, i.e. the total number of neurons in the network (technically also the input values for calculating the first set of gradients, but you will already be storing those, probably for longer), *times* the size of the batch.
  
  Once each batch or mini-batch has had gradients back-propagated, there is no further use for the node/neuron outputs, and the memory allocation can be re-used for the next batch.
  
  > **Comments:**
  > **username_0**: To make sure I understood correctly: we need to store the forward values because 1) we need it to calculate at least the first gradient going back, and 2) for hidden units (for example, sigmoid) the gradient also contains that value in its expression? If yes, then can you direct me towards some proof intuitive or otherwise of the generalizability of the second point above?  
  > **username_1**: @username_0 I added the equation from backprop that uses the value being stored. There are other backprop equations that use the result from that point on.  
  
  ---
  
  