Answers.Body: <p>Most commonly these <strong>states are set to zero</strong>, this usually works, but it can have a negative influence on the performance.</p>
  <p>Another option is to <strong>initialize randomly</strong>, but this is not straight forward and the performance highly depends on the noise level. In <a href="https://scs-europe.net/conf/ecms2015/invited/Contribution_Zimmermann_Grothmann_Tietz.pdf" rel="nofollow noreferrer">this work</a> the noise level is selected in proportion to the prediction error of the first timestep in the series.</p>
  <p>A third option is to <strong>use learnable variables</strong>. You can initialize these randomly and the model will learn the initialization vectors by itself. This is showcased in <a href="https://stackoverflow.com/a/60321446/9891955">this answer on SO</a></p>
  <p>I'd say options one and three are the most straight forward and I've mostly seen the first method.</p>
  <p>For reference, see <a href="https://scs-europe.net/conf/ecms2015/invited/Contribution_Zimmermann_Grothmann_Tietz.pdf" rel="nofollow noreferrer">this paper</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-09T13:42:39.430Z
Answers.Text: Could random initialization make sense if you don't want the LSTM to know where the sequence begins?
Answers.UserDisplayName: Criticize SE actions means ban
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-14T08:28:29.073Z
Answers.Text: I think you cannot really say that in general: The space of the hidden state follows some distribution pattern/manifold that is learned. How exactly that space looks like is up to learning, so starting with the zero-vector can fall perfectly into that hidden space and does not 'appear' as something special to the LSTM, as does random initialization. However, if it is useful for the downstream task to treat the beginning as something special, the hidden space might develop in a way that allows identifying the starting token ....
Answers.UserDisplayName: Chillston
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-14T08:28:31.877Z
Answers.Text: ... I could be wrong but I think you don't really have a lot of influence on that. What do you think about it?
Answers.UserDisplayName: Chillston
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-10T07:59:49.290Z
Answers.LastActivityDate: 2022-05-10T07:59:49.290Z
Answers.UserDisplayName: Chillston
Answers.Body: <p>Your question is related to the initial states of LSTM, where <code>c(t-1)</code> is the cell state (memory) and <code>h(t-1)</code> is the previous LSTM block output.</p>
  <p>As pointed out <a href="https://stats.stackexchange.com/questions/224737/best-way-to-initialize-lstm-state">here</a>, it is reasonable to assume that those are random values.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-15T16:44:58.113Z
Answers.LastActivityDate: 2021-03-15T16:44:58.113Z
Answers.UserDisplayName: penkovsky
Body: <p>Given an LSTM model with 3 cells shown below, what would be the input to the left most cell c(t-1) and h(t-1)?<a href="https://i.sstatic.net/SqL6V.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/SqL6V.png" alt="enter image description here" /></a></p>
  
Tags: <tensorflow><keras><long-short-term-memory>
Title: What is the input to the left most LSTM cell c(t-1) and h(t-1)?
UserDisplayName: Angelina
ThreadText: # What is the input to the left most LSTM cell c(t-1) and h(t-1)?
  
  (Asked by: username_0 on 2021-03-15)
  
  Given an LSTM model with 3 cells shown below, what would be the input to the left most cell c(t-1) and h(t-1)?[![enter image description here](https://i.sstatic.net/SqL6V.png)](https://i.sstatic.net/SqL6V.png)
  
  **Tags:** tensorflow, keras, long-short-term-memory
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2022-05-10. Score: 1
  
  Most commonly these **states are set to zero**, this usually works, but it can have a negative influence on the performance.
  
  Another option is to **initialize randomly**, but this is not straight forward and the performance highly depends on the noise level. In [this work](https://scs-europe.net/conf/ecms2015/invited/Contribution_Zimmermann_Grothmann_Tietz.pdf) the noise level is selected in proportion to the prediction error of the first timestep in the series.
  
  A third option is to **use learnable variables**. You can initialize these randomly and the model will learn the initialization vectors by itself. This is showcased in [this answer on SO](https://stackoverflow.com/a/60321446/9891955)
  
  I'd say options one and three are the most straight forward and I've mostly seen the first method.
  
  For reference, see [this paper](https://scs-europe.net/conf/ecms2015/invited/Contribution_Zimmermann_Grothmann_Tietz.pdf).
  
  > **Comments:**
  > **username_2**: Could random initialization make sense if you don't want the LSTM to know where the sequence begins?  
  > **username_1**: I think you cannot really say that in general: The space of the hidden state follows some distribution pattern/manifold that is learned. How exactly that space looks like is up to learning, so starting with the zero-vector can fall perfectly into that hidden space and does not 'appear' as something special to the LSTM, as does random initialization. However, if it is useful for the downstream task to treat the beginning as something special, the hidden space might develop in a way that allows identifying the starting token ....  
  > **username_1**: ... I could be wrong but I think you don't really have a lot of influence on that. What do you think about it?  
  
  ---
  
  ### Answer by username_3 on 2021-03-15. Score: 0
  
  Your question is related to the initial states of LSTM, where `c(t-1)` is the cell state (memory) and `h(t-1)` is the previous LSTM block output.
  
  As pointed out [here](https://stats.stackexchange.com/questions/224737/best-way-to-initialize-lstm-state), it is reasonable to assume that those are random values.
  
  
  ---
  
  