Answers.Body: <p>Following your notation since the weight <span class="math-container">$w_{11}^2$</span> won't affect <span class="math-container">$z_2^2$</span>, after applying multivariate chain rule it's obvious you don't need the summation term containing it. Therefore your professor's gradient formula w.r.t. <span class="math-container">$w_{11}^2$</span> sounds correct which contains only the <span class="math-container">$z_1^2$</span> summation term.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-05T18:34:54.213Z
Answers.Text: Yes, but $w_{11}^2$ affects $z_1^2$, and $z_1^2$ affects $z_1^3$. So, shouldn't the derivative of $z_1^3$ with respect to $w_{11}^2$ be non-zero and hence, appear in the summation?
Answers.UserDisplayName: Vedant Yadav
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-05T18:41:31.660Z
Answers.Text: @VedantYadav your question is about the loss $e$ which implicitly involves $z^3_1$, so by your own formula above it already appears via the chain rule where the third layer's effect of $z^3_1$ from $w_{11}^2$ is rightly expressed.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-26T04:29:58.757Z
Answers.LastActivityDate: 2024-10-26T04:29:58.757Z
Answers.UserDisplayName: cinch
Body: <p>Some notations for the question: <span class="math-container">$w_{ij}^l$</span> is the weight connecting ith neuron of the layer l to the jth neuron of the layer <span class="math-container">$l-1$</span>. <span class="math-container">$z_i^l$</span> is the activation of ith neuron in the layer l (for simplicity, assume linear activation function). e is the error function.</p>
  <p>My professor of deep learning has written the following expression in gradient calculation:
  <span class="math-container">\begin{equation}
  \nabla_{w_{11}^{2}}e = \frac{\partial e}{\partial z_1^2} \frac{\partial z_1^2}{\partial w_{11}^2}
  \end{equation}</span>.</p>
  <p>However, according to the multivariate chain rule:
  <span class="math-container">$\frac{\partial z}{\partial x_i} = \sum_j \frac{\partial z}{\partial y_j}\frac{\partial y_j}{\partial x_i}$</span></p>
  <p>For example, in calculation of <span class="math-container">$\frac{\partial{e}}{\partial{w_{11}^2}}$</span>, we'll have it equal to <span class="math-container">$\sum_j \frac{\partial{e}}{\partial{z_j}} \frac{\partial{z_j}}{\partial{w_{11}^2}}$</span>. For most <span class="math-container">$z_j$</span>s, the second derivative in the summation is zero, but for the third layer, <span class="math-container">$z_1^3$</span> we'll actually be affected by the change in the weight <span class="math-container">$w_{11}^2$</span>, and hence will have non-zero partial derivative. Am I missing something?</p>
  <p><a href="https://i.sstatic.net/82hyzDAT.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/82hyzDAT.png" alt="enter image description here" /></a></p>
  
Tags: <backpropagation><gradient-descent><multilayer-perceptrons>
Title: Gradient calculation in Backpropogation
UserDisplayName: Vedant Yadav
ThreadText: # Gradient calculation in Backpropogation
  
  (Asked by: username_0 on 2024-10-25)
  
  Some notations for the question: $w\_{ij}^l$ is the weight connecting ith neuron of the layer l to the jth neuron of the layer $l-1$. $z\_i^l$ is the activation of ith neuron in the layer l (for simplicity, assume linear activation function). e is the error function.
  
  My professor of deep learning has written the following expression in gradient calculation:
  \begin{equation}
  \nabla\_{w\_{11}^{2}}e = \frac{\partial e}{\partial z\_1^2} \frac{\partial z\_1^2}{\partial w\_{11}^2}
  \end{equation}.
  
  However, according to the multivariate chain rule:
  $\frac{\partial z}{\partial x\_i} = \sum\_j \frac{\partial z}{\partial y\_j}\frac{\partial y\_j}{\partial x\_i}$
  
  For example, in calculation of $\frac{\partial{e}}{\partial{w\_{11}^2}}$, we'll have it equal to $\sum\_j \frac{\partial{e}}{\partial{z\_j}} \frac{\partial{z\_j}}{\partial{w\_{11}^2}}$. For most $z\_j$s, the second derivative in the summation is zero, but for the third layer, $z\_1^3$ we'll actually be affected by the change in the weight $w\_{11}^2$, and hence will have non-zero partial derivative. Am I missing something?
  
  [![enter image description here](https://i.sstatic.net/82hyzDAT.png)](https://i.sstatic.net/82hyzDAT.png)
  
  **Tags:** backpropagation, gradient-descent, multilayer-perceptrons
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-10-26. Score: 0 | Accepted answer
  
  Following your notation since the weight $w\_{11}^2$ won't affect $z\_2^2$, after applying multivariate chain rule it's obvious you don't need the summation term containing it. Therefore your professor's gradient formula w.r.t. $w\_{11}^2$ sounds correct which contains only the $z\_1^2$ summation term.
  
  > **Comments:**
  > **username_0**: Yes, but $w_{11}^2$ affects $z_1^2$, and $z_1^2$ affects $z_1^3$. So, shouldn't the derivative of $z_1^3$ with respect to $w_{11}^2$ be non-zero and hence, appear in the summation?  
  > **username_1**: @VedantYadav your question is about the loss $e$ which implicitly involves $z^3_1$, so by your own formula above it already appears via the chain rule where the third layer's effect of $z^3_1$ from $w_{11}^2$ is rightly expressed.  
  
  ---
  
  