Answers.Body: <p>This doesn't necessarily answer the question, but it does give some possible solutions to mitigate the problem. </p>
  
  <p>Apparently, the emphasised looping behaviour above is a result of improper initialisation, in which I was initialising with only positive and 0 weights. The looping behavior is largely diminished by proper initialisation, however it is not totally removed. See below for examples:</p>
  
  <pre><code>djbgnuywjkfrrkrkrpueeeeeffrrdv
  kmkkkkkkkkkkkrkkkkkkkkkkkkkkkk
  clfwtpjbsqeeeeeeesssjjeeeeeeee
  ldwgmbvcmmbvcmmmgmmmgmmmgmmmmg
  ywzrfotntntnttntntunuwqzffotnt
  lvvurrrrrrxafllvvurrrrrrxcfhla
  </code></pre>
  
  <p>(These are samples from 6 different uniquely initialised LSTM's on the exact same architecture).</p>
  
  <p>I have also found that this common behaviour is typically combated by adding a certain degree of randomness in feeding in the LSTM's output. This is normally done by interpreting the output at time step <span class="math-container">$t$</span> of the LSTM as a probability distribution, used to pick the next value fed into the LSTM at <span class="math-container">$t+1$</span>. This helps break non-confident looping behaviour which is where looping behaviour most commonly occurs through some quick experimentation, while still mostly retaining confident predictions.</p>
  
  <p>I also tested this by randomising the input a little bit, using a probability distribution where 90% of the time the correct input is selected and 10% a random one is, and back-propagating as normal. This didn't seem to have much of an effect on the looping behaviour, although perhaps might work as a good form of regularisation. I am yet to test it.</p>
  
  <p>You can also use the <em>temperature</em> method in LSTMs, which explained really well here: <a href="https://cs.stackexchange.com/q/79241/20691">https://cs.stackexchange.com/q/79241/20691</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-08T00:42:39.647Z
Answers.Text: I've edited this answer to include the information in the edit of your question. Please, make sure that this answer still makes sense or consistent with your findings.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-07T04:49:30.883Z
Answers.LastActivityDate: 2019-10-08T00:38:40.043Z
Answers.LastEditDate: 2019-10-08T00:38:40.043Z
Answers.UserDisplayName: Recessive
Body: <p>I initialised an LSTM with Xavier initialisation, although I've found this occurs for all initialisations I have tested. When initialised, if the LSTM is tested with a random input, it will get stuck in a cycle, either over a few characters or just one. Example output:</p>
  
  <pre><code>nhhbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
  
  f(b(bf(bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
  
  kk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,m
  </code></pre>
  
  <p>I've also noticed the LSTM is particularly bad in this way, that even when trained it has a tendency to get stuck in loops like this. It seems it has difficulty truly retaining context strong enough to over power the input, activation and output gates with only the forget gate. Is there an explanation for this?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-10-04T20:50:30.017Z
Comments.Text: I think this is a good research question, especially because one may expect that the output of the neural network is random, before training it. So, is this cyclic output due to the specific implementations, the theoretical definition of LSTMs/RNNs and/or the specific initialization of the weights? Of course, in the definition of the RNN, the output at time step $t$ affects the output at a time step $t+1$, but the exact reason behind these cyclic patterns or repetitions in the output isn't very clear.
Comments.UserDisplayName: nbro
Tags: <long-short-term-memory>
Title: Why does an LSTM cycle on initialisation?
UserDisplayName: Recessive
ThreadText: # Why does an LSTM cycle on initialisation?
  
  (Asked by: username_0 on 2019-10-04)
  
  I initialised an LSTM with Xavier initialisation, although I've found this occurs for all initialisations I have tested. When initialised, if the LSTM is tested with a random input, it will get stuck in a cycle, either over a few characters or just one. Example output:
  nhhbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
  
  f(b(bf(bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
  
  kk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,mkk,,m
  
  I've also noticed the LSTM is particularly bad in this way, that even when trained it has a tendency to get stuck in loops like this. It seems it has difficulty truly retaining context strong enough to over power the input, activation and output gates with only the forget gate. Is there an explanation for this?
  
  **Tags:** long-short-term-memory
  > **Comments:**
  > **username_1**: I think this is a good research question, especially because one may expect that the output of the neural network is random, before training it. So, is this cyclic output due to the specific implementations, the theoretical definition of LSTMs/RNNs and/or the specific initialization of the weights? Of course, in the definition of the RNN, the output at time step $t$ affects the output at a time step $t+1$, but the exact reason behind these cyclic patterns or repetitions in the output isn't very clear.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2019-10-07. Score: 0 | Accepted answer
  
  This doesn't necessarily answer the question, but it does give some possible solutions to mitigate the problem. 
  Apparently, the emphasised looping behaviour above is a result of improper initialisation, in which I was initialising with only positive and 0 weights. The looping behavior is largely diminished by proper initialisation, however it is not totally removed. See below for examples:
  djbgnuywjkfrrkrkrpueeeeeffrrdv
  kmkkkkkkkkkkkrkkkkkkkkkkkkkkkk
  clfwtpjbsqeeeeeeesssjjeeeeeeee
  ldwgmbvcmmbvcmmmgmmmgmmmgmmmmg
  ywzrfotntntnttntntunuwqzffotnt
  lvvurrrrrrxafllvvurrrrrrxcfhla
  
  (These are samples from 6 different uniquely initialised LSTM's on the exact same architecture).
  I have also found that this common behaviour is typically combated by adding a certain degree of randomness in feeding in the LSTM's output. This is normally done by interpreting the output at time step $t$ of the LSTM as a probability distribution, used to pick the next value fed into the LSTM at $t+1$. This helps break non-confident looping behaviour which is where looping behaviour most commonly occurs through some quick experimentation, while still mostly retaining confident predictions.
  I also tested this by randomising the input a little bit, using a probability distribution where 90% of the time the correct input is selected and 10% a random one is, and back-propagating as normal. This didn't seem to have much of an effect on the looping behaviour, although perhaps might work as a good form of regularisation. I am yet to test it.
  You can also use the temperature method in LSTMs, which explained really well here: https://cs.stackexchange.com/q/79241/20691.
  
  > **Comments:**
  > **username_1**: I've edited this answer to include the information in the edit of your question. Please, make sure that this answer still makes sense or consistent with your findings.  
  
  ---
  
  