<p>If we write <span class="math-container">$ H^2 = W^{h}H1 + W^{x}X^{2} $</span> then it will be better to understand the backward propagation step.</p>

<p>Now,</p>

<p><span class="math-container">$\frac{\partial}{\partial W^{x}} W^{h}W^{x}X^{1}$</span> can be written as:
<span class="math-container">$\frac{\partial H^2}{\partial H^1}\frac{\partial H^1}{\partial W^{x}} $</span></p>

<p><span class="math-container">$\frac{\partial H^2}{\partial H^1} =  (W^h)^T$</span> and 
<span class="math-container">$\frac{\partial H^1}{\partial W^{x}} = (X^{1})^T $</span></p>

<p>Therefore,</p>

<p><span class="math-container">$\frac{\partial}{\partial W^{x}} W^{h}W^{x}X^{1} = (W^h)^T(X^{1})^T  $</span></p>

<p>I hope it has solved your problem.</p>

CC BY-SA 4.0
2020-05-01T19:29:09.020Z
So, what if we have 3 nodes in the hidden layers (i.e., $H^1$ and $H^2$ should be 3x1 vector) and 2-dimensional inputs ($X^1$ and $X^2$) then $W^h$ is a 3x3 weight matrix and $X^1$ is a 2x1 vector, How can we multiply $(W^h)^T$ and $(X^1)^T$?
user3862410
CC BY-SA 4.0
2020-05-02T03:57:33.410Z
Can you please tell me, in this expression $\frac{\partial}{\partial W^{x}} W^{x}X^{2} = (X^{2})^T$, why have you taken transpose of $(X^2)^T$  as derivative. Why is it not only $(X^2)$
VIJAY
CC BY-SA 4.0
2020-05-01T10:26:52.570Z
2020-05-01T10:26:52.570Z
VIJAY
<p>I think your notations are unclear, but I can give an answer based on what you probably meant. For example, <span class="math-container">$\frac{\partial{L}}{\partial{W^x}}$</span> should be replaced by <span class="math-container">$(\nabla_{W^x_{j:}}L)_{j=1, ...,n}$</span> (assuming everything stays in <span class="math-container">$\mathbb{R}^n$</span>). Also your expression for <span class="math-container">$\frac{\partial{L}}{\partial{W^x}}$</span> is wrong, even accounting for the notation.</p>
<p>Since <span class="math-container">$W^x_{j:}$</span> affects the loss through <span class="math-container">$H_{1,j}$</span> and <span class="math-container">$H_{2,j}$</span>, it would be better to treat the math in this way:
<span class="math-container">$$\nabla_{W^x_{j:}}L=\frac{\partial{L}}{\partial{H_{1,j}}}\nabla_{W^x_{j:}}H_{1,j}+\frac{\partial{L}}{\partial{H_{2,j}}}\nabla_{W^x_{j:}}H_{2,j}$$</span>
Now, <span class="math-container">$H_{1, j}$</span> affects the loss though <span class="math-container">$H_{2,k}\ \forall\ k=1,...,n.$</span> So,
<span class="math-container">$$\frac{\partial{L}}{\partial{H_{1,j}}}=\sum_{k=1}^{n}\frac{\partial{L}} {\partial{H_{2,k}}}W^x_{kj}$$</span>
And,
<span class="math-container">$$\frac{\partial{L}}{\partial{H_{2,j}}}=\sum_{k=1}^{n}\frac{\partial{L}} {\partial{Y_{k}}}W^y_{kj}$$</span>
Similarly, <span class="math-container">$\nabla_YL$</span> can be computed.</p>

CC BY-SA 4.0
2021-03-23T14:24:49.047Z
2021-03-23T14:24:49.047Z
RacingPanda
<p>The product rule of partial derivative:<br>
<span class="math-container">$\frac{\partial}{\partial x} f g = g \frac{\partial}{\partial x} f + f \frac{\partial}{\partial x} g$</span>  </p>

<p>According to this:  <span class="math-container">$\frac{\partial}{\partial W^{x}} W^{h}W^{x}X^{1} = W^{h}X^{1}$</span>, because derivative of other term with respect to <span class="math-container">$W^{x}$</span> is zero. (I am not considering the transpose notation as it depends on how you organize your data.) </p>

<p><strong>However</strong>, Your assumption of giving <span class="math-container">$H^{1}$</span> and <span class="math-container">$X^{2}$</span> as input to second hidden layer is not valid(they are called hidden layer for that reason). The output of first hidden layer (<span class="math-container">$H^{1}$</span>)  will be fed to the input of second hidden layer.
Your output of second hidden layer would be <span class="math-container">$H^{2} = W^{h}  * H^{1}$</span>.  </p>

<p>You have to fed your input <span class="math-container">$X^{1} X^{2}$</span> to your network at once by means of looping or vectorization.</p>

CC BY-SA 4.0
2020-05-01T19:47:02.410Z
Firstly, what if we have 3 nodes in the hidden layers (i.e., $H^1$ and $H^2$ should be 3x1 vector) and 2-dimensional inputs ($X^1$ and $X^2$) then $W^h$ is a 3x3 weight matrix and $X^1$ is a 2x1 vector, How can we multiply $(W^h)^T$ and $(X^1)^T$?

Secondly, think about an [Elman RNN network](http://matlab.izmiran.ru/help/toolbox/nnet/recur94.html). you can think $X^1$ as input at time t=0 and $X^2$ as input at time t=1. Then the $\hat(Y)$ is the output from the time step t=1
user3862410
CC BY-SA 4.0
2020-05-01T20:12:00.297Z
My ans is saying what will be derivative of your asked partial derivative equation. The dimension mismatch is happening because the equation you are getting from forward pass is probably wrong. $H^{2} = W^{h} W^{x} X^{1} + W^{x} X^{2}$ is not right i guess.  
I don't know much about Elman RNN.  My ans is based on simple fully connected N.N as you have mentioned in the question.
** adding a diagram would make people better understand your query. Thanks.
tahiat
CC BY-SA 4.0
2020-05-01T13:23:52.473Z
2020-05-01T13:23:52.473Z
tahiat
<p>I am trying to understand the mathematics behind the forward and backward propagation of neural nets. To make myself more comfortable, I am testing myself with an arbitrarily chosen neural network. However, I am stuck at some point.</p>

<p>Consider a simple fully connected neural network with two hidden layers. For simplicity, choose linear activation function (<span class="math-container">${f(x) = x}$</span>) at all layer. Now consider that this neural network takes two <span class="math-container">$n$</span>-dimensional inputs <span class="math-container">$X^{1}$</span> and <span class="math-container">$X^{2}$</span>. However, the first hidden layer only takes <span class="math-container">$X^1$</span> as the input and produces the output of <span class="math-container">$H^1$</span>. The second hidden layer takes <span class="math-container">$H^{1} $</span>and <span class="math-container">$X^2$</span> as the input and produces the output <span class="math-container">$H^{2}$</span>. The output layer takes <span class="math-container">$H^{2}$</span> as the input and produces the output <span class="math-container">$\hat{Y}$</span>. For simplicity, assume, we do not have any bias.</p>

<p>So, we can write that, <span class="math-container">$H^1 = W^{x1}X^{1}$</span></p>

<p><span class="math-container">$H^2 = W^{h}H1 + W^{x2}X^{2} = W^{h}W^{x1}X^{1} + W^{x2}X^{2}$</span> [substituting the value of <span class="math-container">$H^1$</span>]</p>

<p><span class="math-container">$\hat{Y} = W^{y}H^2$</span></p>

<p>Here, <span class="math-container">$W^{x1}$</span>, <span class="math-container">$W^{x2}$</span>, <span class="math-container">$W^{h}$</span> and <span class="math-container">$W^{y}$</span> are the weight matrix. Now, to make it more interesting, consider a sharing weight matrix <span class="math-container">$W^{x} = W^{x1} = W^{x2}$</span>, which leads, <span class="math-container">$H^1 = W^{x}X^{1}$</span> and <span class="math-container">$H^2 = W^{h}W^{x}X^{1} + W^{x}X^{2}$</span></p>

<p>I do not have any problem to do forward propagation by my hand; however, the problem arises when I tried to make backward propagation and update the <span class="math-container">$W^{x}$</span>.</p>

<p><span class="math-container">$\frac{\partial loss}{\partial W^{x}} = \frac{\partial loss}{\partial H^{2}} . \frac{\partial H^{2}}{\partial W^{x}}$</span> </p>

<p>Substituting, <span class="math-container">$\frac{\partial loss}{\partial H^{2}} = \frac{\partial Y}{\partial H^{2}}. \frac{\partial loss}{\partial Y}$</span> and <span class="math-container">$H^2 = W^{h}W^{x}X^{1} + W^{x}X^{2}$</span></p>

<p><span class="math-container">$\frac{\partial loss}{\partial W^{x}}= \frac{\partial Y}{\partial H^{2}}. \frac{\partial loss}{\partial Y} . \frac{\partial}{\partial W^{x}} (W^{h}W^{x}X^{1} + W^{x}X^{2})$</span></p>

<p>Here I understand that, <span class="math-container">$\frac{\partial Y}{\partial H^{2}} = (W^y)^T$</span> and <span class="math-container">$\frac{\partial}{\partial W^{x}} W^{x}X^{2} = (X^{2})^T$</span> and we can also calculate <span class="math-container">$\frac{\partial Y}{\partial H^{2}}$</span>, if we know the loss function. But how do we calculate <span class="math-container">$\frac{\partial}{\partial W^{x}} W^{h}W^{x}X^{1}$</span>?</p>

<convolutional-neural-networks><backpropagation><weights>
Backpropagation of neural nets with shared weight
user3862410
# Backpropagation of neural nets with shared weight

(Asked by: username_0 on 2020-05-01)

I am trying to understand the mathematics behind the forward and backward propagation of neural nets. To make myself more comfortable, I am testing myself with an arbitrarily chosen neural network. However, I am stuck at some point.

Consider a simple fully connected neural network with two hidden layers. For simplicity, choose linear activation function (${f(x) = x}$) at all layer. Now consider that this neural network takes two $n$-dimensional inputs $X^{1}$ and $X^{2}$. However, the first hidden layer only takes $X^1$ as the input and produces the output of $H^1$. The second hidden layer takes $H^{1} $and $X^2$ as the input and produces the output $H^{2}$. The output layer takes $H^{2}$ as the input and produces the output $\hat{Y}$. For simplicity, assume, we do not have any bias.

So, we can write that, $H^1 = W^{x1}X^{1}$

$H^2 = W^{h}H1 + W^{x2}X^{2} = W^{h}W^{x1}X^{1} + W^{x2}X^{2}$ [substituting the value of $H^1$]

$\hat{Y} = W^{y}H^2$

Here, $W^{x1}$, $W^{x2}$, $W^{h}$ and $W^{y}$ are the weight matrix. Now, to make it more interesting, consider a sharing weight matrix $W^{x} = W^{x1} = W^{x2}$, which leads, $H^1 = W^{x}X^{1}$ and $H^2 = W^{h}W^{x}X^{1} + W^{x}X^{2}$

I do not have any problem to do forward propagation by my hand; however, the problem arises when I tried to make backward propagation and update the $W^{x}$.

$\frac{\partial loss}{\partial W^{x}} = \frac{\partial loss}{\partial H^{2}} . \frac{\partial H^{2}}{\partial W^{x}}$

Substituting, $\frac{\partial loss}{\partial H^{2}} = \frac{\partial Y}{\partial H^{2}}. \frac{\partial loss}{\partial Y}$ and $H^2 = W^{h}W^{x}X^{1} + W^{x}X^{2}$

$\frac{\partial loss}{\partial W^{x}}= \frac{\partial Y}{\partial H^{2}}. \frac{\partial loss}{\partial Y} . \frac{\partial}{\partial W^{x}} (W^{h}W^{x}X^{1} + W^{x}X^{2})$

Here I understand that, $\frac{\partial Y}{\partial H^{2}} = (W^y)^T$ and $\frac{\partial}{\partial W^{x}} W^{x}X^{2} = (X^{2})^T$ and we can also calculate $\frac{\partial Y}{\partial H^{2}}$, if we know the loss function. But how do we calculate $\frac{\partial}{\partial W^{x}} W^{h}W^{x}X^{1}$?

**Tags:** convolutional-neural-networks, backpropagation, weights


---

## 3 answers

### Answer by username_1 on 2020-05-01. Score: 1

If we write $ H^2 = W^{h}H1 + W^{x}X^{2} $ then it will be better to understand the backward propagation step.

Now,

$\frac{\partial}{\partial W^{x}} W^{h}W^{x}X^{1}$ can be written as:
$\frac{\partial H^2}{\partial H^1}\frac{\partial H^1}{\partial W^{x}} $

$\frac{\partial H^2}{\partial H^1} = (W^h)^T$ and
$\frac{\partial H^1}{\partial W^{x}} = (X^{1})^T $

Therefore,

$\frac{\partial}{\partial W^{x}} W^{h}W^{x}X^{1} = (W^h)^T(X^{1})^T $

I hope it has solved your problem.

> **Comments:**
> **username_0**: So, what if we have 3 nodes in the hidden layers (i.e., $H^1$ and $H^2$ should be 3x1 vector) and 2-dimensional inputs ($X^1$ and $X^2$) then $W^h$ is a 3x3 weight matrix and $X^1$ is a 2x1 vector, How can we multiply $(W^h)^T$ and $(X^1)^T$?  
> **username_1**: Can you please tell me, in this expression $\frac{\partial}{\partial W^{x}} W^{x}X^{2} = (X^{2})^T$, why have you taken transpose of $(X^2)^T$  as derivative. Why is it not only $(X^2)$  

---

### Answer by username_2 on 2021-03-23. Score: 1

I think your notations are unclear, but I can give an answer based on what you probably meant. For example, $\frac{\partial{L}}{\partial{W^x}}$ should be replaced by $(\nabla\_{W^x\_{j:}}L)\_{j=1, ...,n}$ (assuming everything stays in $\mathbb{R}^n$). Also your expression for $\frac{\partial{L}}{\partial{W^x}}$ is wrong, even accounting for the notation.

Since $W^x\_{j:}$ affects the loss through $H\_{1,j}$ and $H\_{2,j}$, it would be better to treat the math in this way:
$$\nabla\_{W^x\_{j:}}L=\frac{\partial{L}}{\partial{H\_{1,j}}}\nabla\_{W^x\_{j:}}H\_{1,j}+\frac{\partial{L}}{\partial{H\_{2,j}}}\nabla\_{W^x\_{j:}}H\_{2,j}$$
Now, $H\_{1, j}$ affects the loss though $H\_{2,k}\ \forall\ k=1,...,n.$ So,
$$\frac{\partial{L}}{\partial{H\_{1,j}}}=\sum\_{k=1}^{n}\frac{\partial{L}} {\partial{H\_{2,k}}}W^x\_{kj}$$
And,
$$\frac{\partial{L}}{\partial{H\_{2,j}}}=\sum\_{k=1}^{n}\frac{\partial{L}} {\partial{Y\_{k}}}W^y\_{kj}$$
Similarly, $\nabla\_YL$ can be computed.


---

### Answer by username_3 on 2020-05-01. Score: 0

The product rule of partial derivative:  
$\frac{\partial}{\partial x} f g = g \frac{\partial}{\partial x} f + f \frac{\partial}{\partial x} g$

According to this: $\frac{\partial}{\partial W^{x}} W^{h}W^{x}X^{1} = W^{h}X^{1}$, because derivative of other term with respect to $W^{x}$ is zero. (I am not considering the transpose notation as it depends on how you organize your data.)

**However**, Your assumption of giving $H^{1}$ and $X^{2}$ as input to second hidden layer is not valid(they are called hidden layer for that reason). The output of first hidden layer ($H^{1}$) will be fed to the input of second hidden layer.
Your output of second hidden layer would be $H^{2} = W^{h} \* H^{1}$.

You have to fed your input $X^{1} X^{2}$ to your network at once by means of looping or vectorization.

> **Comments:**
> **username_0**: Firstly, what if we have 3 nodes in the hidden layers (i.e., $H^1$ and $H^2$ should be 3x1 vector) and 2-dimensional inputs ($X^1$ and $X^2$) then $W^h$ is a 3x3 weight matrix and $X^1$ is a 2x1 vector, How can we multiply $(W^h)^T$ and $(X^1)^T$?  Secondly, think about an [Elman RNN network](http://matlab.izmiran.ru/help/toolbox/nnet/recur94.html). you can think $X^1$ as input at time t=0 and $X^2$ as input at time t=1. Then the $\hat(Y)$ is the output from the time step t=1  
> **username_3**: My ans is saying what will be derivative of your asked partial derivative equation. The dimension mismatch is happening because the equation you are getting from forward pass is probably wrong. $H^{2} = W^{h} W^{x} X^{1} + W^{x} X^{2}$ is not right i guess.   I don't know much about Elman RNN.  My ans is based on simple fully connected N.N as you have mentioned in the question. ** adding a diagram would make people better understand your query. Thanks.  

---

