Answers.Body: <p>First I will assume you notate <span class="math-container">$y$</span> as the models output and <span class="math-container">$z$</span> as the ground-truth. Second, I am assuming this is a linear model (No activation functions). Then the gradient math goes as so:</p>
  
  <p><span class="math-container">$
  \begin{align*}
  \frac{dE}{dw_{ij}^1} &amp;= \frac{dE}{dy}\frac{dy}{dw_{ij}^1} \\
  &amp;= \frac{dE}{dy}\sum_k\frac{\partial y}{\partial n_{k}^3}\frac{dn_{k}^3}{dw_{ij}^1} \\
  &amp;= \frac{dE}{dy}\sum_k\frac{\partial y}{\partial n_{k}^3} \frac{\partial n_{k}^3}{ \partial n_{j}^2} \frac{dn_{j}^2}{ dw_{ij}^1}  \\
  &amp;= -2*(z-y)\sum_kw_{kl}^3w_{jk}^2 x_i  \\ 
  \end{align*}
  $</span>  </p>
  
  <p>So the reason you are having trouble to figure out which <span class="math-container">$k$</span> index to use, the answer is because you need to use both and sum over them. The <span class="math-container">$l$</span> index is just the only <span class="math-container">$l$</span> index that exists because you only have one node in that layer.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-29T13:11:35.630Z
Answers.Text: Thank you. I'll have to learn more about derivatives apparently. I would up-vote but I don't have enough reputation, sorry.
Answers.UserDisplayName: Sharpy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-29T13:41:26.557Z
Answers.Text: @Sharpy glad to help and there are tons of online resource going over the differential calculus if you need the refresher, Also if it answered your question, you can still checkmark the answer as correct even if you cant upvote
Answers.UserDisplayName: mshlis
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-29T13:04:40.633Z
Answers.LastActivityDate: 2019-07-29T13:04:40.633Z
Answers.UserDisplayName: mshlis
Body: <p>I've started to learn about neural networks recently and I can't find the answer to this question.</p>
  
  <p>Let's assume there's a neural network (fig. 1)
  <a href="https://i.sstatic.net/LajOf.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/LajOf.png" alt="Figure 1"></a></p>
  
  <p>So if the loss function is:
  <a href="https://i.sstatic.net/K9v5H.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/K9v5H.png" alt="Loss function"></a></p>
  
  <p>and the derivative <a href="https://i.sstatic.net/DxmwA.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/DxmwA.png" alt="deriv"></a> is:
  <a href="https://i.sstatic.net/gqfFl.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/gqfFl.png" alt="Derivative definition"></a></p>
  
  <p>if I want to use this to find <a href="https://i.sstatic.net/hj8Ac.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/hj8Ac.png" alt="dE/dw122"></a> what <em>k</em> and <em>l</em> (well there's only one neuron with index <em>l</em> here, but what if there would be more?) should i use in <a href="https://i.sstatic.net/z1DVl.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/z1DVl.png" alt="w^3_kl"></a> and <a href="https://i.sstatic.net/CyOdM.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/CyOdM.png" alt="w^2_jk"></a>?</p>
  
  <p>I've also found "other" way of backpropagating it's described <a href="https://www.ibu.edu.ba/assets/userfiles/it/2012/eee-ANN-5.pdf" rel="nofollow noreferrer">here</a>, but I can't understand how they came up with that method from the original equation <code>w -= step * dE/dw</code>.</p>
  
  <p>Sorry if I failed to explain my problem. If something isn't clear please ask in comments.</p>
  
Tags: <neural-networks><machine-learning><backpropagation>
Title: What weights should I use while back-propagating?
UserDisplayName: Sharpy
ThreadText: # What weights should I use while back-propagating?
  
  (Asked by: username_0 on 2019-07-29)
  
  I've started to learn about neural networks recently and I can't find the answer to this question.
  
  Let's assume there's a neural network (fig. 1)
  [![Figure 1](https://i.sstatic.net/LajOf.png)](https://i.sstatic.net/LajOf.png)
  
  So if the loss function is:
  [![Loss function](https://i.sstatic.net/K9v5H.png)](https://i.sstatic.net/K9v5H.png)
  
  and the derivative [![deriv](https://i.sstatic.net/DxmwA.png)](https://i.sstatic.net/DxmwA.png) is:
  [![Derivative definition](https://i.sstatic.net/gqfFl.png)](https://i.sstatic.net/gqfFl.png)
  
  if I want to use this to find [![dE/dw122](https://i.sstatic.net/hj8Ac.png)](https://i.sstatic.net/hj8Ac.png) what *k* and *l* (well there's only one neuron with index *l* here, but what if there would be more?) should i use in [![w^3_kl](https://i.sstatic.net/z1DVl.png)](https://i.sstatic.net/z1DVl.png) and [![w^2_jk](https://i.sstatic.net/CyOdM.png)](https://i.sstatic.net/CyOdM.png)?
  
  I've also found "other" way of backpropagating it's described [here](https://www.ibu.edu.ba/assets/userfiles/it/2012/eee-ANN-5.pdf), but I can't understand how they came up with that method from the original equation `w -= step * dE/dw`.
  
  Sorry if I failed to explain my problem. If something isn't clear please ask in comments.
  
  **Tags:** neural-networks, machine-learning, backpropagation
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-07-29. Score: 1 | Accepted answer
  
  First I will assume you notate $y$ as the models output and $z$ as the ground-truth. Second, I am assuming this is a linear model (No activation functions). Then the gradient math goes as so:
  
  $
  \begin{align\*}
  \frac{dE}{dw\_{ij}^1} &= \frac{dE}{dy}\frac{dy}{dw\_{ij}^1} \\
  &= \frac{dE}{dy}\sum\_k\frac{\partial y}{\partial n\_{k}^3}\frac{dn\_{k}^3}{dw\_{ij}^1} \\
  &= \frac{dE}{dy}\sum\_k\frac{\partial y}{\partial n\_{k}^3} \frac{\partial n\_{k}^3}{ \partial n\_{j}^2} \frac{dn\_{j}^2}{ dw\_{ij}^1} \\
  &= -2\*(z-y)\sum\_kw\_{kl}^3w\_{jk}^2 x\_i \\
  \end{align\*}
  $
  
  So the reason you are having trouble to figure out which $k$ index to use, the answer is because you need to use both and sum over them. The $l$ index is just the only $l$ index that exists because you only have one node in that layer.
  
  > **Comments:**
  > **username_0**: Thank you. I'll have to learn more about derivatives apparently. I would up-vote but I don't have enough reputation, sorry.  
  > **username_1**: @Sharpy glad to help and there are tons of online resource going over the differential calculus if you need the refresher, Also if it answered your question, you can still checkmark the answer as correct even if you cant upvote  
  
  ---
  
  