Body: <p>I am trying to understand backpropagation. I used a simple neural network with one input <span class="math-container">$x$</span>, one hidden layer <span class="math-container">$h$</span> and one output layer <span class="math-container">$y$</span>, with weight <span class="math-container">$w_1$</span> connecting <span class="math-container">$x$</span> to <span class="math-container">$h$</span>, and <span class="math-container">$w_2$</span> connecting <span class="math-container">$h$</span> to <span class="math-container">$y$</span></p>
  
  <p><span class="math-container">$$
  x \rightarrow (w_1) \rightarrow h \rightarrow (w_2) \rightarrow y
  $$</span></p>
  
  <p>In my understanding, these are the steps happening while we train a neural network:</p>
  
  <p>The feedforward step.</p>
  
  <p><span class="math-container">\begin{align}
  h=\sigma\left(x w_{1}+b\right)\\
  y^{\prime}=\sigma\left(h w_{2}+b\right)
  \end{align}</span></p>
  
  <p>The loss function.</p>
  
  <p><span class="math-container">$$
  L=\frac{1}{2} \sum\left(y-y^{\prime}\right)^{2}
  $$</span></p>
  
  <p>The gradient calculation</p>
  
  <p><span class="math-container">$$\frac{\partial L}{\partial w_{2}}=\frac{\partial y^{\prime}}{\partial w_{2}} \frac{\partial L}{\partial y^{\prime}}$$</span></p>
  
  <p><span class="math-container">$$\frac{\partial L}{\partial w_{1}}=?$$</span></p>
  
  <p>The weight update</p>
  
  <p><span class="math-container">$$
  w_{i}^{t+1} \leftarrow w_{i}^{t}-\alpha \frac{\partial L}{\partial w_{i}}
  $$</span></p>
  
  <p>I understood most parts of backpropagation, but how do we get the gradients for the middle layer weights <span class="math-container">$dL/dw_1$</span>?</p>
  
  <p>How should we calculate the gradient of a network similar to this?</p>
  
  <p><a href="https://i.sstatic.net/P5fce.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/P5fce.png" alt="enter image description here"></a></p>
  
  <p>Is this the correct equation?</p>
  
  <p><span class="math-container">$$\frac{\partial L}{\partial w_{1}}=\frac{\partial h_{1}}{\partial w_{1}} \frac{\partial w_{7}}{\partial h_{1}} \frac{\partial o_{2}}{\partial w_{7}} \frac{\partial L}{\partial o_{2}}+\frac{\partial h_{1}}{\partial w_{1}} \frac{\partial w_{5}}{\partial h_{1}} \frac{\partial o_{1}}{\partial w_{5}} \frac{\partial L}{\partial o_{1}}$$</span></p>
  
Tags: <neural-networks><backpropagation><gradient-descent>
Title: How is the gradient calculated for the middle layer's weights?
UserDisplayName: Eka
ThreadText: # How is the gradient calculated for the middle layer's weights?
  
  (Asked by: username_0 on 2018-03-08)
  
  I am trying to understand backpropagation. I used a simple neural network with one input $x$, one hidden layer $h$ and one output layer $y$, with weight $w\_1$ connecting $x$ to $h$, and $w\_2$ connecting $h$ to $y$
  
  $$
  x \rightarrow (w\_1) \rightarrow h \rightarrow (w\_2) \rightarrow y
  $$
  
  In my understanding, these are the steps happening while we train a neural network:
  
  The feedforward step.
  
  \begin{align}
  h=\sigma\left(x w\_{1}+b\right)\\
  y^{\prime}=\sigma\left(h w\_{2}+b\right)
  \end{align}
  
  The loss function.
  
  $$
  L=\frac{1}{2} \sum\left(y-y^{\prime}\right)^{2}
  $$
  
  The gradient calculation
  
  $$\frac{\partial L}{\partial w\_{2}}=\frac{\partial y^{\prime}}{\partial w\_{2}} \frac{\partial L}{\partial y^{\prime}}$$
  
  $$\frac{\partial L}{\partial w\_{1}}=?$$
  
  The weight update
  
  $$
  w\_{i}^{t+1} \leftarrow w\_{i}^{t}-\alpha \frac{\partial L}{\partial w\_{i}}
  $$
  
  I understood most parts of backpropagation, but how do we get the gradients for the middle layer weights $dL/dw\_1$?
  
  How should we calculate the gradient of a network similar to this?
  
  [![enter image description here](https://i.sstatic.net/P5fce.png)](https://i.sstatic.net/P5fce.png)
  
  Is this the correct equation?
  
  $$\frac{\partial L}{\partial w\_{1}}=\frac{\partial h\_{1}}{\partial w\_{1}} \frac{\partial w\_{7}}{\partial h\_{1}} \frac{\partial o\_{2}}{\partial w\_{7}} \frac{\partial L}{\partial o\_{2}}+\frac{\partial h\_{1}}{\partial w\_{1}} \frac{\partial w\_{5}}{\partial h\_{1}} \frac{\partial o\_{1}}{\partial w\_{5}} \frac{\partial L}{\partial o\_{1}}$$
  
  **Tags:** neural-networks, backpropagation, gradient-descent
  
  ---
  
  ## 0 answers
  
  