Answers.Body: <blockquote>
    <p>Why do we update all layers simultaneously while training a neural network?</p>
  </blockquote>
  
  <p>We typically train a neural network with gradient descent and back-propagation. Gradient descent is the iterative algorithm used to update the parameters and back-propagation is the algorithm used to compute the gradient of the loss function with respect to each of these parameters.</p>
  
  <p>Let's denote a vector that contains all learnable parameters of a neural network <span class="math-container">$M$</span> by <span class="math-container">$\mathbf{w} = \left[w_1, \dots, w_n \right] \in \mathbb{R}^n$</span> (so <span class="math-container">$M$</span> contains <span class="math-container">$n$</span> learnable parameters), the loss function of <span class="math-container">$M$</span> by <span class="math-container">$\mathcal{L}$</span>,  the gradient of the loss function with respect to each parameter <span class="math-container">$w_i$</span> of <span class="math-container">$M$</span> by <span class="math-container">$ \nabla \mathcal{L} = \left[ \frac{\partial \mathcal{L}}{\partial w_1}, \dots, \frac{\partial \mathcal{L}}{\partial w_n}  \right] \in \mathbb{R}^n$</span>, then the gradient descent step to update all parameters is</p>
  
  <p><span class="math-container">\begin{align}
  \mathbf{w} \leftarrow \mathbf{w} - \gamma * \nabla \mathcal{L} \tag{1} \label{1}
  \end{align}</span></p>
  
  <p>where <span class="math-container">$\gamma \in \mathbb{R}$</span> is the learning rate.</p>
  
  <p>In equation \ref{1}, we are assigning to <span class="math-container">$\mathbf{w}$</span> the value <span class="math-container">$\mathbf{w} - \gamma * \nabla \mathcal{L}$</span>, so we are updating all parameters <span class="math-container">$\mathbf{w}$</span> simultaneously, so we are also updating all layers simultaneously. </p>
  
  <p>In principle, you could update each parameter <span class="math-container">$w_i$</span> individually. To be more precise, you would have the following update rule</p>
  
  <p><span class="math-container">\begin{align}
  w_i \leftarrow w_i - \gamma * \frac{\partial \mathcal{L}}{\partial w_i}, \; \forall i\tag{2} \label{2}
  \end{align}</span></p>
  
  <p>So, you could update first <span class="math-container">$w_1$</span>, then <span class="math-container">$w_2$</span>, and so on. </p>
  
  <p>Actually, you don't need to update the parameters sequentially (also because there is no real order of the parameters). You can actually update them in any other way. You can update the parameters in any way because, although the computation of the gradient highly depends on the structure of the neural network (so if you change the structure, the computation of the gradient also changes), once the gradient is computed, you already have all information to update each parameter independently of each other. </p>
  
  <p>You typically update all parameters (or layers) simultaneously because, in practice, you work with vectors and matrices rather than scalars in order to take benefit from efficient matrix multiplication algorithms and hardware (i.e. GPUs).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-18T19:17:50.290Z
Answers.Text: After got the gradient $ \nabla \mathcal{L} = \left[ \frac{\partial \mathcal{L}}{\partial w_1}, \dots, \frac{\partial \mathcal{L}}{\partial w_n}  \right] $, the BP algorithm should update weights $w_1, w_2, ... w_n$ at next step. For example, after update $w_n$ , the $\frac{∂L}{∂w_{n-1}}$ may not still the best "direction" for update $w_{n-1}$. I'm not sure am I correct. If true, why not re-calculate the partial derivative from $w_{n-1}$ to $w_1$ then update $w_{n-1}$?
Answers.UserDisplayName: huang
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-16T12:05:40.867Z
Answers.LastActivityDate: 2020-04-16T12:10:42.197Z
Answers.LastEditDate: 2020-04-16T12:10:42.197Z
Answers.UserDisplayName: nbro
Body: <blockquote>
    <p>Very deep models involve the composition of several functions or layers. The gradient tells how to update each parameter, under the assumption that the other layers do not change. In practice, we update all of the layers simultaneously.</p>
  </blockquote>
  
  <p>The above is an extract from Ian Goodfellow's Deep Learning  - which talks about the need for batch normalization. </p>
  
  <p><strong>Why do we update all the layers simultaneously?</strong> Instead, if we update layers one at a time during backpropagation - it will eliminate the need for batch normalization, right?</p>
  
  <p>Reference: <a href="https://machinelearningmastery.com/batch-normalization-for-training-of-deep-neural-networks/" rel="noreferrer">A Gentle Introduction to Batch Normalization for Deep Neural Networks</a></p>
  
  <p>P.S. The attached link says: <em>Because all layers are changed during an update, the update procedure is forever chasing a moving target</em>. Apart from the main question, it would be great if someone could explain why exactly a <em>moving target</em> is being referred to in the above sentence.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-05T13:48:10.103Z
Comments.Text: Two years after this question, Google has now published a paper about doing backpropagation layer-by-layer, caching the "target" outputs of each layers. It does indeed seem a lot more efficient: https://ai.googleblog.com/2022/07/enhancing-backpropagation-via-local.html
Comments.UserDisplayName: Narrateur du chaos
Tags: <neural-networks><deep-learning><backpropagation><gradient-descent><batch-normalization>
Title: Why do we update all layers simultaneously while training a neural network?
UserDisplayName: stoic-santiago
ThreadText: # Why do we update all layers simultaneously while training a neural network?
  
  (Asked by: username_0 on 2020-04-16)
  
  > Very deep models involve the composition of several functions or layers. The gradient tells how to update each parameter, under the assumption that the other layers do not change. In practice, we update all of the layers simultaneously.
  
  The above is an extract from Ian Goodfellow's Deep Learning - which talks about the need for batch normalization.
  
  **Why do we update all the layers simultaneously?** Instead, if we update layers one at a time during backpropagation - it will eliminate the need for batch normalization, right?
  
  Reference: [A Gentle Introduction to Batch Normalization for Deep Neural Networks](https://machinelearningmastery.com/batch-normalization-for-training-of-deep-neural-networks/)
  
  P.S. The attached link says: *Because all layers are changed during an update, the update procedure is forever chasing a moving target*. Apart from the main question, it would be great if someone could explain why exactly a *moving target* is being referred to in the above sentence.
  
  **Tags:** neural-networks, deep-learning, backpropagation, gradient-descent, batch-normalization
  
  > **Comments:**
  > **username_1**: Two years after this question, Google has now published a paper about doing backpropagation layer-by-layer, caching the "target" outputs of each layers. It does indeed seem a lot more efficient: https://ai.googleblog.com/2022/07/enhancing-backpropagation-via-local.html  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-04-16. Score: 4 | Accepted answer
  
  > Why do we update all layers simultaneously while training a neural network?
  
  We typically train a neural network with gradient descent and back-propagation. Gradient descent is the iterative algorithm used to update the parameters and back-propagation is the algorithm used to compute the gradient of the loss function with respect to each of these parameters.
  
  Let's denote a vector that contains all learnable parameters of a neural network $M$ by $\mathbf{w} = \left[w\_1, \dots, w\_n \right] \in \mathbb{R}^n$ (so $M$ contains $n$ learnable parameters), the loss function of $M$ by $\mathcal{L}$, the gradient of the loss function with respect to each parameter $w\_i$ of $M$ by $ \nabla \mathcal{L} = \left[ \frac{\partial \mathcal{L}}{\partial w\_1}, \dots, \frac{\partial \mathcal{L}}{\partial w\_n} \right] \in \mathbb{R}^n$, then the gradient descent step to update all parameters is
  
  \begin{align}
  \mathbf{w} \leftarrow \mathbf{w} - \gamma \* \nabla \mathcal{L} \tag{1} \label{1}
  \end{align}
  
  where $\gamma \in \mathbb{R}$ is the learning rate.
  
  In equation \ref{1}, we are assigning to $\mathbf{w}$ the value $\mathbf{w} - \gamma \* \nabla \mathcal{L}$, so we are updating all parameters $\mathbf{w}$ simultaneously, so we are also updating all layers simultaneously.
  
  In principle, you could update each parameter $w\_i$ individually. To be more precise, you would have the following update rule
  
  \begin{align}
  w\_i \leftarrow w\_i - \gamma \* \frac{\partial \mathcal{L}}{\partial w\_i}, \; \forall i\tag{2} \label{2}
  \end{align}
  
  So, you could update first $w\_1$, then $w\_2$, and so on.
  
  Actually, you don't need to update the parameters sequentially (also because there is no real order of the parameters). You can actually update them in any other way. You can update the parameters in any way because, although the computation of the gradient highly depends on the structure of the neural network (so if you change the structure, the computation of the gradient also changes), once the gradient is computed, you already have all information to update each parameter independently of each other.
  
  You typically update all parameters (or layers) simultaneously because, in practice, you work with vectors and matrices rather than scalars in order to take benefit from efficient matrix multiplication algorithms and hardware (i.e. GPUs).
  
  > **Comments:**
  > **username_3**: After got the gradient $ \nabla \mathcal{L} = \left[ \frac{\partial \mathcal{L}}{\partial w_1}, \dots, \frac{\partial \mathcal{L}}{\partial w_n}  \right] $, the BP algorithm should update weights $w_1, w_2, ... w_n$ at next step. For example, after update $w_n$ , the $\frac{∂L}{∂w_{n-1}}$ may not still the best "direction" for update $w_{n-1}$. I'm not sure am I correct. If true, why not re-calculate the partial derivative from $w_{n-1}$ to $w_1$ then update $w_{n-1}$?  
  
  ---
  
  