Answers.Body: <blockquote>
    <p>Is the gradient at a layer (of a feed-forward neural network) independent of the activations of the previous layers?</p>
  </blockquote>
  
  <p>Yes, as per @recessive answer they are indeed independent of the previous layers.</p>
  
  <p>The goal of back-propagation is to trace the loss(error between target and network output) to specific weights in the network, and then tweak them to minimize this loss.
  For this to be possible, the activation must be independent of previous activation functions(going forward in the network).</p>
  
  <p>It is very helpful to have a good understanding of the back-propagation process when reading these papers, and I personally suggest watching this video to get a good understanding of it:</p>
  
  <p><a href="https://www.youtube.com/watch?v=Ilg3gGewQ5U" rel="nofollow noreferrer">https://www.youtube.com/watch?v=Ilg3gGewQ5U</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-22T13:58:57.510Z
Answers.LastActivityDate: 2019-10-22T13:58:57.510Z
Answers.UserDisplayName: Krrrl
Answers.Body: <p>Yes, this is the premise of back-propagation, the gradient at layer <span class="math-container">$j_{n}$</span> is not impacted by the gradient at layer <span class="math-container">$j_{n-1}$</span>. This allows you to start with a gradient at the output layer and propagated it back through the network to the input layer.</p>
  
  <p>It is however impacted by the gradient at <span class="math-container">$j_{n+1}$</span>, which the back-prop algorithm also follows.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-11T05:33:48.960Z
Answers.LastActivityDate: 2019-10-22T01:16:11.403Z
Answers.LastEditDate: 2019-10-22T01:16:11.403Z
Answers.UserDisplayName: Recessive
Body: <p>Is the gradient at a layer (of a feed-forward neural network) independent of the activations of the previous layers?</p>
  
  <p>I read this in a paper titled <a href="https://arxiv.org/abs/1712.08969" rel="nofollow noreferrer">Mean Field Residual Networks: On the Edge of Chaos</a> (2017). I am not sure how far this is true, because the error depends on those activations.</p>
  
Tags: <neural-networks><deep-learning><backpropagation><gradient-descent><gradient>
Title: Is the gradient at a layer independent of the activations of the previous layers?
UserDisplayName: Snehal Reddy
ThreadText: # Is the gradient at a layer independent of the activations of the previous layers?
  
  (Asked by: username_0 on 2019-10-11)
  
  Is the gradient at a layer (of a feed-forward neural network) independent of the activations of the previous layers?
  
  I read this in a paper titled [Mean Field Residual Networks: On the Edge of Chaos](https://arxiv.org/abs/1712.08969) (2017). I am not sure how far this is true, because the error depends on those activations.
  
  **Tags:** neural-networks, deep-learning, backpropagation, gradient-descent, gradient
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2019-10-22. Score: 2
  
  > Is the gradient at a layer (of a feed-forward neural network) independent of the activations of the previous layers?
  
  Yes, as per @username_2 answer they are indeed independent of the previous layers.
  
  The goal of back-propagation is to trace the loss(error between target and network output) to specific weights in the network, and then tweak them to minimize this loss.
  For this to be possible, the activation must be independent of previous activation functions(going forward in the network).
  
  It is very helpful to have a good understanding of the back-propagation process when reading these papers, and I personally suggest watching this video to get a good understanding of it:
  
  <https://www.youtube.com/watch?v=Ilg3gGewQ5U>
  
  
  ---
  
  ### Answer by username_2 on 2019-10-11. Score: 2
  
  Yes, this is the premise of back-propagation, the gradient at layer $j\_{n}$ is not impacted by the gradient at layer $j\_{n-1}$. This allows you to start with a gradient at the output layer and propagated it back through the network to the input layer.
  
  It is however impacted by the gradient at $j\_{n+1}$, which the back-prop algorithm also follows.
  
  
  ---
  
  