<p>I am trying to understand backpropagation. I used a simple neural network with one input <span class="math-container">$x$</span>, one hidden layer <span class="math-container">$h$</span> and one output layer <span class="math-container">$y$</span>, with weight <span class="math-container">$w_1$</span> connecting <span class="math-container">$x$</span> to <span class="math-container">$h$</span>, and <span class="math-container">$w_2$</span> connecting <span class="math-container">$h$</span> to <span class="math-container">$y$</span></p>

<p><span class="math-container">$$
x \rightarrow (w_1) \rightarrow h \rightarrow (w_2) \rightarrow y
$$</span></p>

<p>In my understanding, these are the steps happening while we train a neural network:</p>

<p>The feedforward step.</p>

<p><span class="math-container">\begin{align}
h=\sigma\left(x w_{1}+b\right)\\
y^{\prime}=\sigma\left(h w_{2}+b\right)
\end{align}</span></p>

<p>The loss function.</p>

<p><span class="math-container">$$
L=\frac{1}{2} \sum\left(y-y^{\prime}\right)^{2}
$$</span></p>

<p>The gradient calculation</p>

<p><span class="math-container">$$\frac{\partial L}{\partial w_{2}}=\frac{\partial y^{\prime}}{\partial w_{2}} \frac{\partial L}{\partial y^{\prime}}$$</span></p>

<p><span class="math-container">$$\frac{\partial L}{\partial w_{1}}=?$$</span></p>

<p>The weight update</p>

<p><span class="math-container">$$
w_{i}^{t+1} \leftarrow w_{i}^{t}-\alpha \frac{\partial L}{\partial w_{i}}
$$</span></p>

<p>I understood most parts of backpropagation, but how do we get the gradients for the middle layer weights <span class="math-container">$dL/dw_1$</span>?</p>

<p>How should we calculate the gradient of a network similar to this?</p>

<p><a href="https://i.sstatic.net/P5fce.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/P5fce.png" alt="enter image description here"></a></p>

<p>Is this the correct equation?</p>

<p><span class="math-container">$$\frac{\partial L}{\partial w_{1}}=\frac{\partial h_{1}}{\partial w_{1}} \frac{\partial w_{7}}{\partial h_{1}} \frac{\partial o_{2}}{\partial w_{7}} \frac{\partial L}{\partial o_{2}}+\frac{\partial h_{1}}{\partial w_{1}} \frac{\partial w_{5}}{\partial h_{1}} \frac{\partial o_{1}}{\partial w_{5}} \frac{\partial L}{\partial o_{1}}$$</span></p>

<neural-networks><backpropagation><gradient-descent>
How is the gradient calculated for the middle layer's weights?
Eka
# How is the gradient calculated for the middle layer's weights?

(Asked by: username_0 on 2018-03-08)

I am trying to understand backpropagation. I used a simple neural network with one input $x$, one hidden layer $h$ and one output layer $y$, with weight $w\_1$ connecting $x$ to $h$, and $w\_2$ connecting $h$ to $y$

$$
x \rightarrow (w\_1) \rightarrow h \rightarrow (w\_2) \rightarrow y
$$

In my understanding, these are the steps happening while we train a neural network:

The feedforward step.

\begin{align}
h=\sigma\left(x w\_{1}+b\right)\\
y^{\prime}=\sigma\left(h w\_{2}+b\right)
\end{align}

The loss function.

$$
L=\frac{1}{2} \sum\left(y-y^{\prime}\right)^{2}
$$

The gradient calculation

$$\frac{\partial L}{\partial w\_{2}}=\frac{\partial y^{\prime}}{\partial w\_{2}} \frac{\partial L}{\partial y^{\prime}}$$

$$\frac{\partial L}{\partial w\_{1}}=?$$

The weight update

$$
w\_{i}^{t+1} \leftarrow w\_{i}^{t}-\alpha \frac{\partial L}{\partial w\_{i}}
$$

I understood most parts of backpropagation, but how do we get the gradients for the middle layer weights $dL/dw\_1$?

How should we calculate the gradient of a network similar to this?

[![enter image description here](https://i.sstatic.net/P5fce.png)](https://i.sstatic.net/P5fce.png)

Is this the correct equation?

$$\frac{\partial L}{\partial w\_{1}}=\frac{\partial h\_{1}}{\partial w\_{1}} \frac{\partial w\_{7}}{\partial h\_{1}} \frac{\partial o\_{2}}{\partial w\_{7}} \frac{\partial L}{\partial o\_{2}}+\frac{\partial h\_{1}}{\partial w\_{1}} \frac{\partial w\_{5}}{\partial h\_{1}} \frac{\partial o\_{1}}{\partial w\_{5}} \frac{\partial L}{\partial o\_{1}}$$

**Tags:** neural-networks, backpropagation, gradient-descent


---

## 0 answers

