Answers.Body: <p>The long-short term memory (LSTM) is a type of recurrent neural network, which is <em>only</em> suited for sequence modelling, that is, to keep track of statistical dependencies between elements of a sequence. </p>
  
  <p>The LSTM prediction capabilities are limited to the training data that is used to train it, the <a href="https://en.wikipedia.org/wiki/Inductive_bias" rel="nofollow noreferrer">inductive bias</a> (in the case of LSTMs, the inductive bias particularly refers to the fact that elements of a sequence are dependent on each other) and the available computation resources. However, storytelling often assumes the existence of a <a href="https://en.wikipedia.org/wiki/Commonsense_knowledge_(artificial_intelligence)" rel="nofollow noreferrer">common-sense knowledge</a> between the storyteller and the listener (but LSTM completely ignores this) and requires the true understanding of language, which is believed to be an <a href="https://ai.stackexchange.com/a/13262/2444">AI-complete problem</a> (in simple words, it is a very complex task, which probably <a href="https://arxiv.org/abs/1901.02918" rel="nofollow noreferrer">cannot be exactly solved with statistical models</a>).</p>
  
  <p>Furthermore, even though LSTM <em>partially</em> addresses the vanishing gradient problem (and they were specifically created to partially solve this issue), they can still suffer from the <a href="https://stats.stackexchange.com/q/320919/82135">exploding gradient problem</a>. Moreover, although LSTMs and, in general, neural networks are universal function approximators, in practice, functions may not be continuous, which is an assumption made in the universal approximation theorems.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-29T13:05:45.950Z
Answers.Text: Just a minor comment: it is probably worth noting that exploding gradients can be fixed easily with gradient clipping in most cases.
Answers.UserDisplayName: Mathias Müller
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-29T13:09:17.537Z
Answers.Text: @MathiasMüller Thanks for your feedback and thanks for contributing to the site! I've actually not worked much with LSTMs (even though I am quite familiar with the theory), so your feedback is definitely valuable (especially, if you had hands-on experience with LSTMs). Clearly, I am aware of gradient clipping, but I am not sure how much it can help in this specific case, because I am not sure I've ever used it (in this case).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-29T13:46:35.313Z
Answers.Text: Do you mean, this specific task of telling a story in particular? I don't know, I don't work on story generation. Some [technical reading about exploding gradients in RNNs](https://arxiv.org/pdf/1211.5063.pdf) that you might not be aware of. The authors argue that LSTMs fix vanishing gradients to some extent, but they (= LSTMs) do not fix exploding gradients. The authors propose clipping the gradient norm.
Answers.UserDisplayName: Mathias Müller
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-29T13:54:28.127Z
Answers.Text: @MathiasMüller No, I didn't mean storytelling in particular. I had already come across this paper, but I think I've never fully read it. Thanks for pointing this out. If I have some time, I will have a deeper look at it ;)
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-08T17:31:11.503Z
Answers.LastActivityDate: 2019-11-08T17:45:52.827Z
Answers.LastEditDate: 2019-11-08T17:45:52.827Z
Answers.UserDisplayName: nbro
Body: <p>There is a recent trend in people using LSTMs to write novels. I haven’t attempted this myself. From what I’m hearing, they can tell a story, but it seems they lose the context of the story rather quickly. After which they begin constructing new, but not necessarily related constructs. </p>
  
  <p>Can they construct a plot in the long term?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-11-08T23:50:00.353Z
Comments.Text: An LSTM has trouble remembering over very long time frames. It also, as nbro said, can't really learn common-sense knowledge. It means it can't truly determine what information is important to remember (like a character dying for example) consistently. It very easily loses context when you let it run for extended periods of time (especially if it's longer then the input it was trained on), so it can't realistically write a complete story coherently. Even if you just wanted a short story (~200 words), it is very likely it would lose some important contextual knowledge.
Comments.UserDisplayName: Recessive
Tags: <deep-learning><recurrent-neural-networks><long-short-term-memory><applications>
Title: Why can't LSTMs tell a long story?
UserDisplayName: junfanbl
ThreadText: # Why can't LSTMs tell a long story?
  
  (Asked by: username_0 on 2019-11-08)
  
  There is a recent trend in people using LSTMs to write novels. I haven’t attempted this myself. From what I’m hearing, they can tell a story, but it seems they lose the context of the story rather quickly. After which they begin constructing new, but not necessarily related constructs.
  
  Can they construct a plot in the long term?
  
  **Tags:** deep-learning, recurrent-neural-networks, long-short-term-memory, applications
  
  > **Comments:**
  > **username_1**: An LSTM has trouble remembering over very long time frames. It also, as username_2 said, can't really learn common-sense knowledge. It means it can't truly determine what information is important to remember (like a character dying for example) consistently. It very easily loses context when you let it run for extended periods of time (especially if it's longer then the input it was trained on), so it can't realistically write a complete story coherently. Even if you just wanted a short story (~200 words), it is very likely it would lose some important contextual knowledge.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-11-08. Score: 3
  
  The long-short term memory (LSTM) is a type of recurrent neural network, which is *only* suited for sequence modelling, that is, to keep track of statistical dependencies between elements of a sequence.
  
  The LSTM prediction capabilities are limited to the training data that is used to train it, the [inductive bias](https://en.wikipedia.org/wiki/Inductive_bias) (in the case of LSTMs, the inductive bias particularly refers to the fact that elements of a sequence are dependent on each other) and the available computation resources. However, storytelling often assumes the existence of a [common-sense knowledge](https://en.wikipedia.org/wiki/Commonsense_knowledge_(artificial_intelligence)) between the storyteller and the listener (but LSTM completely ignores this) and requires the true understanding of language, which is believed to be an [AI-complete problem](https://ai.stackexchange.com/a/13262/2444) (in simple words, it is a very complex task, which probably [cannot be exactly solved with statistical models](https://arxiv.org/abs/1901.02918)).
  
  Furthermore, even though LSTM *partially* addresses the vanishing gradient problem (and they were specifically created to partially solve this issue), they can still suffer from the [exploding gradient problem](https://stats.stackexchange.com/q/320919/82135). Moreover, although LSTMs and, in general, neural networks are universal function approximators, in practice, functions may not be continuous, which is an assumption made in the universal approximation theorems.
  
  > **Comments:**
  > **username_3**: Just a minor comment: it is probably worth noting that exploding gradients can be fixed easily with gradient clipping in most cases.  
  > **username_2**: @username_3 Thanks for your feedback and thanks for contributing to the site! I've actually not worked much with LSTMs (even though I am quite familiar with the theory), so your feedback is definitely valuable (especially, if you had hands-on experience with LSTMs). Clearly, I am aware of gradient clipping, but I am not sure how much it can help in this specific case, because I am not sure I've ever used it (in this case).  
  > **username_3**: Do you mean, this specific task of telling a story in particular? I don't know, I don't work on story generation. Some [technical reading about exploding gradients in RNNs](https://arxiv.org/pdf/1211.5063.pdf) that you might not be aware of. The authors argue that LSTMs fix vanishing gradients to some extent, but they (= LSTMs) do not fix exploding gradients. The authors propose clipping the gradient norm.  
  > **username_2**: @username_3 No, I didn't mean storytelling in particular. I had already come across this paper, but I think I've never fully read it. Thanks for pointing this out. If I have some time, I will have a deeper look at it ;)  
  
  ---
  
  