Answers.Body: <p>(1) $X_t$ and $H_{t-1}$ are concatenated. The blog you cited explained its notation "<em>Lines merging denote concatenation</em>". For example, if $X_t=[1,2,3]$ and $H_{t-1}=[4,5,6,7]$, then their concatenation is $[1,2,3,4,5,6,7]$</p>
  
  <p>(2) When you say "<em>input weights</em>" or "<em>weights of the input of the previous time step</em>", are you referring to the $W_i$ in your cited blog? If so they are not <em>the weights of the connections between the time-steps/units</em>. They are part of the input gate only. The <em>connections between the time-steps/units</em> do not have weights applied to them.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-03T10:26:12.270Z
Answers.Text: Thanks for the answer. But: If on every time step, more elements are appended, doesn't the vector get huge after many time steps? I mean, there is the forget, ignore and update gate, but as far as I know, they only set the values they want to block to zero, they don't erase the value from the vector.
Answers.UserDisplayName: Ben
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-03T17:39:13.197Z
Answers.Text: So each gate is essentially a small NN, just collapsed down into one unit using matrix multiplication?
Answers.UserDisplayName: Ben
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-03T18:20:28.980Z
Answers.Text: @Ben In each step $X_t$ and $H_{t-1}$ is fixed size, so it won't become longer and longer. Note that in each cell, the concatenation of $H_{t-1}$ and $X_t$ are not directly connected to $H_{t}$, but they go through a $\sigma$ which "collapse down" the size. And yes, here $\sigma$ is small NN.
Answers.UserDisplayName: user12075
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-02T18:30:59.620Z
Answers.LastActivityDate: 2018-09-02T18:30:59.620Z
Answers.UserDisplayName: user12075
Body: <p>I am currently looking into LSTMs. I found <a href="http://colah.github.io/posts/2015-08-Understanding-LSTMs/" rel="nofollow noreferrer">this nice blog post</a>, which is already very helpful, but still, there are things I don't understand, mostly because of the collapsed layers. </p>
  
  <ul>
  <li>The input $X_t$, and the output of the previous time step $H_{t-1}$, how do they get combined? Multiplied, added or what? </li>
  <li>The input weights and the weights of the input of the previous time step, those are just the weights of the connections between the time-steps/units, right?</li>
  </ul>
  
Tags: <machine-learning><deep-learning><long-short-term-memory><architecture>
Title: How do the current input and the output of the previous time step get combined in an LSTM?
UserDisplayName: Ben
ThreadText: # How do the current input and the output of the previous time step get combined in an LSTM?
  
  (Asked by: username_0 on 2018-09-02)
  
  I am currently looking into LSTMs. I found [this nice blog post](http://colah.github.io/posts/2015-08-Understanding-LSTMs/), which is already very helpful, but still, there are things I don't understand, mostly because of the collapsed layers.
  
  * The input $X\_t$, and the output of the previous time step $H\_{t-1}$, how do they get combined? Multiplied, added or what?
  * The input weights and the weights of the input of the previous time step, those are just the weights of the connections between the time-steps/units, right?
  
  **Tags:** machine-learning, deep-learning, long-short-term-memory, architecture
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-09-02. Score: 1 | Accepted answer
  
  (1) $X\_t$ and $H\_{t-1}$ are concatenated. The blog you cited explained its notation "*Lines merging denote concatenation*". For example, if $X\_t=[1,2,3]$ and $H\_{t-1}=[4,5,6,7]$, then their concatenation is $[1,2,3,4,5,6,7]$
  
  (2) When you say "*input weights*" or "*weights of the input of the previous time step*", are you referring to the $W\_i$ in your cited blog? If so they are not *the weights of the connections between the time-steps/units*. They are part of the input gate only. The *connections between the time-steps/units* do not have weights applied to them.
  
  > **Comments:**
  > **username_0**: Thanks for the answer. But: If on every time step, more elements are appended, doesn't the vector get huge after many time steps? I mean, there is the forget, ignore and update gate, but as far as I know, they only set the values they want to block to zero, they don't erase the value from the vector.  
  > **username_0**: So each gate is essentially a small NN, just collapsed down into one unit using matrix multiplication?  
  > **username_1**: @username_0 In each step $X_t$ and $H_{t-1}$ is fixed size, so it won't become longer and longer. Note that in each cell, the concatenation of $H_{t-1}$ and $X_t$ are not directly connected to $H_{t}$, but they go through a $\sigma$ which "collapse down" the size. And yes, here $\sigma$ is small NN.  
  
  ---
  
  