Answers.Body: <p>If we write <span class="math-container">$ H^2 = W^{h}H1 + W^{x}X^{2} $</span> then it will be better to understand the backward propagation step.</p>
  
  <p>Now,</p>
  
  <p><span class="math-container">$\frac{\partial}{\partial W^{x}} W^{h}W^{x}X^{1}$</span> can be written as:
  <span class="math-container">$\frac{\partial H^2}{\partial H^1}\frac{\partial H^1}{\partial W^{x}} $</span></p>
  
  <p><span class="math-container">$\frac{\partial H^2}{\partial H^1} =  (W^h)^T$</span> and 
  <span class="math-container">$\frac{\partial H^1}{\partial W^{x}} = (X^{1})^T $</span></p>
  
  <p>Therefore,</p>
  
  <p><span class="math-container">$\frac{\partial}{\partial W^{x}} W^{h}W^{x}X^{1} = (W^h)^T(X^{1})^T  $</span></p>
  
  <p>I hope it has solved your problem.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-01T19:29:09.020Z
Answers.Text: So, what if we have 3 nodes in the hidden layers (i.e., $H^1$ and $H^2$ should be 3x1 vector) and 2-dimensional inputs ($X^1$ and $X^2$) then $W^h$ is a 3x3 weight matrix and $X^1$ is a 2x1 vector, How can we multiply $(W^h)^T$ and $(X^1)^T$?
Answers.UserDisplayName: user3862410
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-02T03:57:33.410Z
Answers.Text: Can you please tell me, in this expression $\frac{\partial}{\partial W^{x}} W^{x}X^{2} = (X^{2})^T$, why have you taken transpose of $(X^2)^T$  as derivative. Why is it not only $(X^2)$
Answers.UserDisplayName: VIJAY
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-01T10:26:52.570Z
Answers.LastActivityDate: 2020-05-01T10:26:52.570Z
Answers.UserDisplayName: VIJAY
Answers.Body: <p>I think your notations are unclear, but I can give an answer based on what you probably meant. For example, <span class="math-container">$\frac{\partial{L}}{\partial{W^x}}$</span> should be replaced by <span class="math-container">$(\nabla_{W^x_{j:}}L)_{j=1, ...,n}$</span> (assuming everything stays in <span class="math-container">$\mathbb{R}^n$</span>). Also your expression for <span class="math-container">$\frac{\partial{L}}{\partial{W^x}}$</span> is wrong, even accounting for the notation.</p>
  <p>Since <span class="math-container">$W^x_{j:}$</span> affects the loss through <span class="math-container">$H_{1,j}$</span> and <span class="math-container">$H_{2,j}$</span>, it would be better to treat the math in this way:
  <span class="math-container">$$\nabla_{W^x_{j:}}L=\frac{\partial{L}}{\partial{H_{1,j}}}\nabla_{W^x_{j:}}H_{1,j}+\frac{\partial{L}}{\partial{H_{2,j}}}\nabla_{W^x_{j:}}H_{2,j}$$</span>
  Now, <span class="math-container">$H_{1, j}$</span> affects the loss though <span class="math-container">$H_{2,k}\ \forall\ k=1,...,n.$</span> So,
  <span class="math-container">$$\frac{\partial{L}}{\partial{H_{1,j}}}=\sum_{k=1}^{n}\frac{\partial{L}} {\partial{H_{2,k}}}W^x_{kj}$$</span>
  And,
  <span class="math-container">$$\frac{\partial{L}}{\partial{H_{2,j}}}=\sum_{k=1}^{n}\frac{\partial{L}} {\partial{Y_{k}}}W^y_{kj}$$</span>
  Similarly, <span class="math-container">$\nabla_YL$</span> can be computed.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-23T14:24:49.047Z
Answers.LastActivityDate: 2021-03-23T14:24:49.047Z
Answers.UserDisplayName: RacingPanda
Answers.Body: <p>The product rule of partial derivative:<br>
  <span class="math-container">$\frac{\partial}{\partial x} f g = g \frac{\partial}{\partial x} f + f \frac{\partial}{\partial x} g$</span>  </p>
  
  <p>According to this:  <span class="math-container">$\frac{\partial}{\partial W^{x}} W^{h}W^{x}X^{1} = W^{h}X^{1}$</span>, because derivative of other term with respect to <span class="math-container">$W^{x}$</span> is zero. (I am not considering the transpose notation as it depends on how you organize your data.) </p>
  
  <p><strong>However</strong>, Your assumption of giving <span class="math-container">$H^{1}$</span> and <span class="math-container">$X^{2}$</span> as input to second hidden layer is not valid(they are called hidden layer for that reason). The output of first hidden layer (<span class="math-container">$H^{1}$</span>)  will be fed to the input of second hidden layer.
  Your output of second hidden layer would be <span class="math-container">$H^{2} = W^{h}  * H^{1}$</span>.  </p>
  
  <p>You have to fed your input <span class="math-container">$X^{1} X^{2}$</span> to your network at once by means of looping or vectorization.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-01T19:47:02.410Z
Answers.Text: Firstly, what if we have 3 nodes in the hidden layers (i.e., $H^1$ and $H^2$ should be 3x1 vector) and 2-dimensional inputs ($X^1$ and $X^2$) then $W^h$ is a 3x3 weight matrix and $X^1$ is a 2x1 vector, How can we multiply $(W^h)^T$ and $(X^1)^T$?
  
  Secondly, think about an [Elman RNN network](http://matlab.izmiran.ru/help/toolbox/nnet/recur94.html). you can think $X^1$ as input at time t=0 and $X^2$ as input at time t=1. Then the $\hat(Y)$ is the output from the time step t=1
Answers.UserDisplayName: user3862410
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-01T20:12:00.297Z
Answers.Text: My ans is saying what will be derivative of your asked partial derivative equation. The dimension mismatch is happening because the equation you are getting from forward pass is probably wrong. $H^{2} = W^{h} W^{x} X^{1} + W^{x} X^{2}$ is not right i guess.  
  I don't know much about Elman RNN.  My ans is based on simple fully connected N.N as you have mentioned in the question.
  ** adding a diagram would make people better understand your query. Thanks.
Answers.UserDisplayName: tahiat
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-01T13:23:52.473Z
Answers.LastActivityDate: 2020-05-01T13:23:52.473Z
Answers.UserDisplayName: tahiat
Body: <p>I am trying to understand the mathematics behind the forward and backward propagation of neural nets. To make myself more comfortable, I am testing myself with an arbitrarily chosen neural network. However, I am stuck at some point.</p>
  
  <p>Consider a simple fully connected neural network with two hidden layers. For simplicity, choose linear activation function (<span class="math-container">${f(x) = x}$</span>) at all layer. Now consider that this neural network takes two <span class="math-container">$n$</span>-dimensional inputs <span class="math-container">$X^{1}$</span> and <span class="math-container">$X^{2}$</span>. However, the first hidden layer only takes <span class="math-container">$X^1$</span> as the input and produces the output of <span class="math-container">$H^1$</span>. The second hidden layer takes <span class="math-container">$H^{1} $</span>and <span class="math-container">$X^2$</span> as the input and produces the output <span class="math-container">$H^{2}$</span>. The output layer takes <span class="math-container">$H^{2}$</span> as the input and produces the output <span class="math-container">$\hat{Y}$</span>. For simplicity, assume, we do not have any bias.</p>
  
  <p>So, we can write that, <span class="math-container">$H^1 = W^{x1}X^{1}$</span></p>
  
  <p><span class="math-container">$H^2 = W^{h}H1 + W^{x2}X^{2} = W^{h}W^{x1}X^{1} + W^{x2}X^{2}$</span> [substituting the value of <span class="math-container">$H^1$</span>]</p>
  
  <p><span class="math-container">$\hat{Y} = W^{y}H^2$</span></p>
  
  <p>Here, <span class="math-container">$W^{x1}$</span>, <span class="math-container">$W^{x2}$</span>, <span class="math-container">$W^{h}$</span> and <span class="math-container">$W^{y}$</span> are the weight matrix. Now, to make it more interesting, consider a sharing weight matrix <span class="math-container">$W^{x} = W^{x1} = W^{x2}$</span>, which leads, <span class="math-container">$H^1 = W^{x}X^{1}$</span> and <span class="math-container">$H^2 = W^{h}W^{x}X^{1} + W^{x}X^{2}$</span></p>
  
  <p>I do not have any problem to do forward propagation by my hand; however, the problem arises when I tried to make backward propagation and update the <span class="math-container">$W^{x}$</span>.</p>
  
  <p><span class="math-container">$\frac{\partial loss}{\partial W^{x}} = \frac{\partial loss}{\partial H^{2}} . \frac{\partial H^{2}}{\partial W^{x}}$</span> </p>
  
  <p>Substituting, <span class="math-container">$\frac{\partial loss}{\partial H^{2}} = \frac{\partial Y}{\partial H^{2}}. \frac{\partial loss}{\partial Y}$</span> and <span class="math-container">$H^2 = W^{h}W^{x}X^{1} + W^{x}X^{2}$</span></p>
  
  <p><span class="math-container">$\frac{\partial loss}{\partial W^{x}}= \frac{\partial Y}{\partial H^{2}}. \frac{\partial loss}{\partial Y} . \frac{\partial}{\partial W^{x}} (W^{h}W^{x}X^{1} + W^{x}X^{2})$</span></p>
  
  <p>Here I understand that, <span class="math-container">$\frac{\partial Y}{\partial H^{2}} = (W^y)^T$</span> and <span class="math-container">$\frac{\partial}{\partial W^{x}} W^{x}X^{2} = (X^{2})^T$</span> and we can also calculate <span class="math-container">$\frac{\partial Y}{\partial H^{2}}$</span>, if we know the loss function. But how do we calculate <span class="math-container">$\frac{\partial}{\partial W^{x}} W^{h}W^{x}X^{1}$</span>?</p>
  
Tags: <convolutional-neural-networks><backpropagation><weights>
Title: Backpropagation of neural nets with shared weight
UserDisplayName: user3862410
ThreadText: # Backpropagation of neural nets with shared weight
  
  (Asked by: username_0 on 2020-05-01)
  
  I am trying to understand the mathematics behind the forward and backward propagation of neural nets. To make myself more comfortable, I am testing myself with an arbitrarily chosen neural network. However, I am stuck at some point.
  
  Consider a simple fully connected neural network with two hidden layers. For simplicity, choose linear activation function (${f(x) = x}$) at all layer. Now consider that this neural network takes two $n$-dimensional inputs $X^{1}$ and $X^{2}$. However, the first hidden layer only takes $X^1$ as the input and produces the output of $H^1$. The second hidden layer takes $H^{1} $and $X^2$ as the input and produces the output $H^{2}$. The output layer takes $H^{2}$ as the input and produces the output $\hat{Y}$. For simplicity, assume, we do not have any bias.
  
  So, we can write that, $H^1 = W^{x1}X^{1}$
  
  $H^2 = W^{h}H1 + W^{x2}X^{2} = W^{h}W^{x1}X^{1} + W^{x2}X^{2}$ [substituting the value of $H^1$]
  
  $\hat{Y} = W^{y}H^2$
  
  Here, $W^{x1}$, $W^{x2}$, $W^{h}$ and $W^{y}$ are the weight matrix. Now, to make it more interesting, consider a sharing weight matrix $W^{x} = W^{x1} = W^{x2}$, which leads, $H^1 = W^{x}X^{1}$ and $H^2 = W^{h}W^{x}X^{1} + W^{x}X^{2}$
  
  I do not have any problem to do forward propagation by my hand; however, the problem arises when I tried to make backward propagation and update the $W^{x}$.
  
  $\frac{\partial loss}{\partial W^{x}} = \frac{\partial loss}{\partial H^{2}} . \frac{\partial H^{2}}{\partial W^{x}}$
  
  Substituting, $\frac{\partial loss}{\partial H^{2}} = \frac{\partial Y}{\partial H^{2}}. \frac{\partial loss}{\partial Y}$ and $H^2 = W^{h}W^{x}X^{1} + W^{x}X^{2}$
  
  $\frac{\partial loss}{\partial W^{x}}= \frac{\partial Y}{\partial H^{2}}. \frac{\partial loss}{\partial Y} . \frac{\partial}{\partial W^{x}} (W^{h}W^{x}X^{1} + W^{x}X^{2})$
  
  Here I understand that, $\frac{\partial Y}{\partial H^{2}} = (W^y)^T$ and $\frac{\partial}{\partial W^{x}} W^{x}X^{2} = (X^{2})^T$ and we can also calculate $\frac{\partial Y}{\partial H^{2}}$, if we know the loss function. But how do we calculate $\frac{\partial}{\partial W^{x}} W^{h}W^{x}X^{1}$?
  
  **Tags:** convolutional-neural-networks, backpropagation, weights
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2020-05-01. Score: 1
  
  If we write $ H^2 = W^{h}H1 + W^{x}X^{2} $ then it will be better to understand the backward propagation step.
  
  Now,
  
  $\frac{\partial}{\partial W^{x}} W^{h}W^{x}X^{1}$ can be written as:
  $\frac{\partial H^2}{\partial H^1}\frac{\partial H^1}{\partial W^{x}} $
  
  $\frac{\partial H^2}{\partial H^1} = (W^h)^T$ and
  $\frac{\partial H^1}{\partial W^{x}} = (X^{1})^T $
  
  Therefore,
  
  $\frac{\partial}{\partial W^{x}} W^{h}W^{x}X^{1} = (W^h)^T(X^{1})^T $
  
  I hope it has solved your problem.
  
  > **Comments:**
  > **username_0**: So, what if we have 3 nodes in the hidden layers (i.e., $H^1$ and $H^2$ should be 3x1 vector) and 2-dimensional inputs ($X^1$ and $X^2$) then $W^h$ is a 3x3 weight matrix and $X^1$ is a 2x1 vector, How can we multiply $(W^h)^T$ and $(X^1)^T$?  
  > **username_1**: Can you please tell me, in this expression $\frac{\partial}{\partial W^{x}} W^{x}X^{2} = (X^{2})^T$, why have you taken transpose of $(X^2)^T$  as derivative. Why is it not only $(X^2)$  
  
  ---
  
  ### Answer by username_2 on 2021-03-23. Score: 1
  
  I think your notations are unclear, but I can give an answer based on what you probably meant. For example, $\frac{\partial{L}}{\partial{W^x}}$ should be replaced by $(\nabla\_{W^x\_{j:}}L)\_{j=1, ...,n}$ (assuming everything stays in $\mathbb{R}^n$). Also your expression for $\frac{\partial{L}}{\partial{W^x}}$ is wrong, even accounting for the notation.
  
  Since $W^x\_{j:}$ affects the loss through $H\_{1,j}$ and $H\_{2,j}$, it would be better to treat the math in this way:
  $$\nabla\_{W^x\_{j:}}L=\frac{\partial{L}}{\partial{H\_{1,j}}}\nabla\_{W^x\_{j:}}H\_{1,j}+\frac{\partial{L}}{\partial{H\_{2,j}}}\nabla\_{W^x\_{j:}}H\_{2,j}$$
  Now, $H\_{1, j}$ affects the loss though $H\_{2,k}\ \forall\ k=1,...,n.$ So,
  $$\frac{\partial{L}}{\partial{H\_{1,j}}}=\sum\_{k=1}^{n}\frac{\partial{L}} {\partial{H\_{2,k}}}W^x\_{kj}$$
  And,
  $$\frac{\partial{L}}{\partial{H\_{2,j}}}=\sum\_{k=1}^{n}\frac{\partial{L}} {\partial{Y\_{k}}}W^y\_{kj}$$
  Similarly, $\nabla\_YL$ can be computed.
  
  
  ---
  
  ### Answer by username_3 on 2020-05-01. Score: 0
  
  The product rule of partial derivative:  
  $\frac{\partial}{\partial x} f g = g \frac{\partial}{\partial x} f + f \frac{\partial}{\partial x} g$
  
  According to this: $\frac{\partial}{\partial W^{x}} W^{h}W^{x}X^{1} = W^{h}X^{1}$, because derivative of other term with respect to $W^{x}$ is zero. (I am not considering the transpose notation as it depends on how you organize your data.)
  
  **However**, Your assumption of giving $H^{1}$ and $X^{2}$ as input to second hidden layer is not valid(they are called hidden layer for that reason). The output of first hidden layer ($H^{1}$) will be fed to the input of second hidden layer.
  Your output of second hidden layer would be $H^{2} = W^{h} \* H^{1}$.
  
  You have to fed your input $X^{1} X^{2}$ to your network at once by means of looping or vectorization.
  
  > **Comments:**
  > **username_0**: Firstly, what if we have 3 nodes in the hidden layers (i.e., $H^1$ and $H^2$ should be 3x1 vector) and 2-dimensional inputs ($X^1$ and $X^2$) then $W^h$ is a 3x3 weight matrix and $X^1$ is a 2x1 vector, How can we multiply $(W^h)^T$ and $(X^1)^T$?  Secondly, think about an [Elman RNN network](http://matlab.izmiran.ru/help/toolbox/nnet/recur94.html). you can think $X^1$ as input at time t=0 and $X^2$ as input at time t=1. Then the $\hat(Y)$ is the output from the time step t=1  
  > **username_3**: My ans is saying what will be derivative of your asked partial derivative equation. The dimension mismatch is happening because the equation you are getting from forward pass is probably wrong. $H^{2} = W^{h} W^{x} X^{1} + W^{x} X^{2}$ is not right i guess.   I don't know much about Elman RNN.  My ans is based on simple fully connected N.N as you have mentioned in the question. ** adding a diagram would make people better understand your query. Thanks.  
  
  ---
  
  