Answers.Body: <p>My understanding is that you have posted the last equation on <code>p. 19</code> in the dissertation (please correct me if I'm wrong). </p>
  
  <p>The derivation is indeed for a single LSTM layer as <em>e<sub>k</sub>(t)</em> is the error at the output (the assumed network topology is also mentioned just below <code>Eq. (2.8)</code> in the dissertation):</p>
  
  <blockquote>
    <p>Finally, assuming a layered network topology with a standard input layer, a hidden layer consisting of memory blocks, and a standard output layer[...]</p>
  </blockquote>
  
  <p>Let's assume now that there are two stacked LSTM layers. In this case, the derivation in the dissertation applies to the last hidden LSTM layer (the one below the output layer), where <code>Eqs. (3.10) -- (3.12)</code> give you the partial derivatives for the weights at each gate for a cell in that layer. To derive the deltas for the hidden LSTM layer <em>below</em>, you have to compute the partial derivatives with respect to the portions of <em>net<sub>c<sub>v</sub><sup>j</sup></sub>(t)</em>, <em>net<sub>in<sub>j</sub></sub>(t)</em> and <em>net<sub>f<sub>j</sub></sub>(t)</em> terms corresponding to the <em>outputs</em> of the preceding hidden LSTM layer, and then use those in the same way you used <em>e<sub>k</sub>(t)</em> for the current LSTM layer. Underneath all that unappealing notation is just the usual multi-layer backprop rule (well, with the truncated RTRL twist). If you stack more LSTM layers, just keep propagating the errors further down through the respective gates until you reach the input layer.</p>
  
  <p>For a slightly more intuitive explanation, if you look at <code>Fig. 2.1</code> in the dissertation, you can assume that in a multi-layered LSTM the <code>IN</code> in fact includes the <code>OUT</code> of the preceding LSTM layer.</p>
  
  <h2>Edit</h2>
  
  <p>There is a nice diagram of the flow of partial derivatives <a href="https://arunmallya.github.io/writeups/nn/lstm/index.html#/7" rel="nofollow noreferrer">here</a> (also see subsequent slides).</p>
  
  <p>In this example, all <em>x<sup>t</sup></em> terms represent the external input to that layer at time step <em>t</em>. In a multi-layer LSTM, this includes input from the LSTM layer below the current one. To propagate the error to the layer below, take the derivatives with respect to all gate weights <strong>corresponding to <em>x<sup>t</sup></em></strong> and apply the chain rule.</p>
  
  <p>The reason why the derivation for the output gate weights is different is that the gating is applied <em>after</em> the cell state is updated, whereas the cell, input and forget gates are applied <em>before</em> that. This matters when computing the gradients.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-10T11:34:20.593Z
Answers.LastActivityDate: 2018-07-16T01:34:26.387Z
Answers.LastEditDate: 2018-07-16T01:34:26.387Z
Answers.UserDisplayName: cantordust
Body: <p>I am currently trying to understand the mathematics in Ger's paper <a href="http://www.felixgers.de/papers/phd.pdf" rel="nofollow noreferrer">Long Short-Term Memory in Recurrent Neural Networks</a>. I have found the document clear and readable so far. </p>
  
  <p>On pg. 21 of the pdf (pg. 13 of the paper), he derives the backward pass equations for output gates. He writes</p>
  
  <p><span class="math-container">$$\frac{\partial y^k(t)}{\partial y_{out_{j}}} e_k(t) = h(s_{c_{j}^{v}}(t)) w_{k c_{j}^{v}} \delta_{k}(t)$$</span>.</p>
  
  <p>If we replaced <span class="math-container">$\delta_{k}(t)$</span>, the expression becomes</p>
  
  <p><span class="math-container">$$\frac{\partial y^k(t)}{\partial y_{out_{j}}} e_k(t) = h(s_{c_{j}^{v}}(t)) w_{k c_{j}^{v}} f'(net_k(t)) e_k(t)$$</span>.</p>
  
  <p>He states that the result of the partial derivative <span class="math-container">$\frac{\partial y^k(t)}{\partial y_{out_{j}}}$</span> comes from differentiating the forward pass equations for the output units. </p>
  
  <p>From that and from the inclusion of <span class="math-container">$e_k(t)$</span>, the paper implies that there is only one hidden LSTM layer. If there are multiple hidden LSTM layers, it wouldn't make sense. </p>
  
  <blockquote class="spoiler">
    <p> Because if <span class="math-container">$k$</span> is the index of LSTM cells that the current cell is outputting to, then <span class="math-container">$e_k(t)$</span> would not exist since the cell output isn't compared with the target output of the network. And if <span class="math-container">$k$</span> is the index of output neurons, then <span class="math-container">$w_{k c_{j}^{v}}$</span> would not exist since the memory cells are not directly connected to output neurons. And <span class="math-container">$k$</span> cannot mean different things since both components are placed under a sum over <span class="math-container">$k$</span>. Therefore, it only makes sense if the paper assumes a single LSTM layer.</p>
  </blockquote>
  
  <p>So, how would one modify the backward pass derivation steps for an LSTM layer that outputs to another LSTM layer? </p>
  
Tags: <neural-networks><machine-learning><recurrent-neural-networks><backpropagation><long-short-term-memory>
Title: How to change the backward pass for an LSTM layer that outputs to another LSTM layer?
UserDisplayName: Angle Qian
ThreadText: # How to change the backward pass for an LSTM layer that outputs to another LSTM layer?
  
  (Asked by: username_0 on 2018-07-09)
  
  I am currently trying to understand the mathematics in Ger's paper [Long Short-Term Memory in Recurrent Neural Networks](http://www.felixgers.de/papers/phd.pdf). I have found the document clear and readable so far.
  
  On pg. 21 of the pdf (pg. 13 of the paper), he derives the backward pass equations for output gates. He writes
  
  $$\frac{\partial y^k(t)}{\partial y\_{out\_{j}}} e\_k(t) = h(s\_{c\_{j}^{v}}(t)) w\_{k c\_{j}^{v}} \delta\_{k}(t)$$.
  
  If we replaced $\delta\_{k}(t)$, the expression becomes
  
  $$\frac{\partial y^k(t)}{\partial y\_{out\_{j}}} e\_k(t) = h(s\_{c\_{j}^{v}}(t)) w\_{k c\_{j}^{v}} f'(net\_k(t)) e\_k(t)$$.
  
  He states that the result of the partial derivative $\frac{\partial y^k(t)}{\partial y\_{out\_{j}}}$ comes from differentiating the forward pass equations for the output units.
  
  From that and from the inclusion of $e\_k(t)$, the paper implies that there is only one hidden LSTM layer. If there are multiple hidden LSTM layers, it wouldn't make sense.
  
  > Because if $k$ is the index of LSTM cells that the current cell is outputting to, then $e\_k(t)$ would not exist since the cell output isn't compared with the target output of the network. And if $k$ is the index of output neurons, then $w\_{k c\_{j}^{v}}$ would not exist since the memory cells are not directly connected to output neurons. And $k$ cannot mean different things since both components are placed under a sum over $k$. Therefore, it only makes sense if the paper assumes a single LSTM layer.
  
  So, how would one modify the backward pass derivation steps for an LSTM layer that outputs to another LSTM layer?
  
  **Tags:** neural-networks, machine-learning, recurrent-neural-networks, backpropagation, long-short-term-memory
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-07-10. Score: 2 | Accepted answer
  
  My understanding is that you have posted the last equation on `p. 19` in the dissertation (please correct me if I'm wrong).
  
  The derivation is indeed for a single LSTM layer as *ek(t)* is the error at the output (the assumed network topology is also mentioned just below `Eq. (2.8)` in the dissertation):
  
  > Finally, assuming a layered network topology with a standard input layer, a hidden layer consisting of memory blocks, and a standard output layer[...]
  
  Let's assume now that there are two stacked LSTM layers. In this case, the derivation in the dissertation applies to the last hidden LSTM layer (the one below the output layer), where `Eqs. (3.10) -- (3.12)` give you the partial derivatives for the weights at each gate for a cell in that layer. To derive the deltas for the hidden LSTM layer *below*, you have to compute the partial derivatives with respect to the portions of *netcvj(t)*, *netinj(t)* and *netfj(t)* terms corresponding to the *outputs* of the preceding hidden LSTM layer, and then use those in the same way you used *ek(t)* for the current LSTM layer. Underneath all that unappealing notation is just the usual multi-layer backprop rule (well, with the truncated RTRL twist). If you stack more LSTM layers, just keep propagating the errors further down through the respective gates until you reach the input layer.
  
  For a slightly more intuitive explanation, if you look at `Fig. 2.1` in the dissertation, you can assume that in a multi-layered LSTM the `IN` in fact includes the `OUT` of the preceding LSTM layer.
  
  ## Edit
  
  There is a nice diagram of the flow of partial derivatives [here](https://arunmallya.github.io/writeups/nn/lstm/index.html#/7) (also see subsequent slides).
  
  In this example, all *xt* terms represent the external input to that layer at time step *t*. In a multi-layer LSTM, this includes input from the LSTM layer below the current one. To propagate the error to the layer below, take the derivatives with respect to all gate weights **corresponding to *xt*** and apply the chain rule.
  
  The reason why the derivation for the output gate weights is different is that the gating is applied *after* the cell state is updated, whereas the cell, input and forget gates are applied *before* that. This matters when computing the gradients.
  
  
  ---
  
  