Answers.Body: <p>The answer is implied in the term "backpropagation". All gradients are calculated at the same time. That is, the error from your loss function is propagated backwards from the output and through your whole network. This propagation results in an error associated with each weight in the network, which determines how you change each weight in order to minimize your loss function.</p>
  
  <blockquote>
    <p>or do we allow <span class="math-container">$V$</span> to update along with <span class="math-container">$W$</span>?</p>
  </blockquote>
  
  <p>Yes. This saves time, since many of the results of intermediate computations used to update <span class="math-container">$V$</span> can be reused in the update of <span class="math-container">$W$</span>. </p>
  
  <p>See <a href="http://neuralnetworksanddeeplearning.com" rel="nofollow noreferrer">http://neuralnetworksanddeeplearning.com</a> for a detailed description of the backpropagation algorithm.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-04T00:15:33.513Z
Answers.LastActivityDate: 2019-04-04T01:18:07.873Z
Answers.LastEditDate: 2019-04-04T01:18:07.873Z
Answers.UserDisplayName: Philip Raeisghasem
Body: <p>I am new to Deep Learning.</p>
  <p>Suppose that we have a neural network with one input layer, one output layer, and one hidden layer. Let's refer to the weights from input to hidden as <span class="math-container">$W$</span> and the weights from hidden to output as <span class="math-container">$V$</span>. Suppose that we have initialized <span class="math-container">$W$</span> and <span class="math-container">$V$</span>, and ran them through the neural network via the forward algorithm/pass. Suppose that we have updated <span class="math-container">$V$</span> via backpropagation.</p>
  <p>When estimating the ideal weights for <span class="math-container">$W$</span>, do we keep the weights <span class="math-container">$V$</span> constant when updating <span class="math-container">$W$</span> via gradient descent given we already calculated <span class="math-container">$V$</span>, or do we allow <span class="math-container">$V$</span> to update along with <span class="math-container">$W$</span>?</p>
  <p>So, in the code, which I am trying to do from scratch, do we include <span class="math-container">$V$</span> in the for loop that will be used for gradient descent to find <span class="math-container">$W$</span>? In other words, do we simply use the same <span class="math-container">$V$</span> for every iteration of gradient descent?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-04T15:00:31.760Z
Comments.Text: If you're interested in the details of backpropagation and general review about deep learning, you may take a look at the following technical report on arXiv:
  [Deep learning for pedestrians: backpropagation in CNNs](https://arxiv.org/abs/1811.11987)
  Make sure to download locally and open with Acrobat reader to enjoy the illustrations!
Comments.UserDisplayName: ranlot
Tags: <neural-networks><deep-learning><backpropagation><gradient-descent>
Title: Does backpropagation update weights one layer at a time?
UserDisplayName: Joshua Jones
ThreadText: # Does backpropagation update weights one layer at a time?
  
  (Asked by: username_0 on 2019-04-03)
  
  I am new to Deep Learning.
  
  Suppose that we have a neural network with one input layer, one output layer, and one hidden layer. Let's refer to the weights from input to hidden as $W$ and the weights from hidden to output as $V$. Suppose that we have initialized $W$ and $V$, and ran them through the neural network via the forward algorithm/pass. Suppose that we have updated $V$ via backpropagation.
  
  When estimating the ideal weights for $W$, do we keep the weights $V$ constant when updating $W$ via gradient descent given we already calculated $V$, or do we allow $V$ to update along with $W$?
  
  So, in the code, which I am trying to do from scratch, do we include $V$ in the for loop that will be used for gradient descent to find $W$? In other words, do we simply use the same $V$ for every iteration of gradient descent?
  
  **Tags:** neural-networks, deep-learning, backpropagation, gradient-descent
  
  > **Comments:**
  > **username_1**: If you're interested in the details of backpropagation and general review about deep learning, you may take a look at the following technical report on arXiv: [Deep learning for pedestrians: backpropagation in CNNs](https://arxiv.org/abs/1811.11987) Make sure to download locally and open with Acrobat reader to enjoy the illustrations!  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-04-04. Score: 2 | Accepted answer
  
  The answer is implied in the term "backpropagation". All gradients are calculated at the same time. That is, the error from your loss function is propagated backwards from the output and through your whole network. This propagation results in an error associated with each weight in the network, which determines how you change each weight in order to minimize your loss function.
  
  > or do we allow $V$ to update along with $W$?
  
  Yes. This saves time, since many of the results of intermediate computations used to update $V$ can be reused in the update of $W$.
  
  See <http://neuralnetworksanddeeplearning.com> for a detailed description of the backpropagation algorithm.
  
  
  ---
  
  