Answers.Body: <p>A.) The algorithm can only learn patterns from the data, so if you want it to align to a specific functional form of language, it would make sense to curate your data to only contain sentences of that very structure. </p>
  
  <p>If you tune your hyper-parameters such that the model is not as complex as the default, it should work for datasets that are smaller. But this relies on the observation that the thing you are trying to learn isn't as complex (as opposed to learning from a large corpus of Nietzsche).</p>
  
  <p>B.) What is happening is that you are <em>overfitting the data</em>, such that the LSTM isn't generalizing to your intended goal. In essence, overfitting means that your model is learning irrelevant details that <em>by chance</em> happen to predict the intended goal in the training data. To alleviate this, use a validation set.</p>
  
  <p>And finally (but very importantly), these machine learning techniques don't learn the same way as humans do. To understand why you'd need to learn the math behind NNs, and I can't think of an intuitive explanation as to why.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-04-24T16:14:25.250Z
Answers.LastActivityDate: 2017-04-24T16:14:25.250Z
Answers.UserDisplayName: k.c. sayz 'k.c sayz'
Answers.Body: <p>LSTMs are good and everything but you can try some exotic methods like Convolutional Neural Network for NLP. Perhaps it would be more suited for this need. Have a look at this.</p>
  
  <ol>
  <li><a href="http://www.wildml.com/2016/01/attention-and-memory-in-deep-learning-and-nlp/" rel="nofollow noreferrer">http://www.wildml.com/2016/01/attention-and-memory-in-deep-learning-and-nlp/</a></li>
  <li><a href="http://www.wildml.com/2015/12/implementing-a-cnn-for-text-classification-in-tensorflow/" rel="nofollow noreferrer">http://www.wildml.com/2015/12/implementing-a-cnn-for-text-classification-in-tensorflow/</a></li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-04-25T14:00:05.233Z
Answers.LastActivityDate: 2017-04-25T14:00:05.233Z
Answers.UserDisplayName: Souradeep Nanda
Body: <p>I'm playing with an LSTM to generate text. In particular, this one:</p>
  
  <p><a href="https://raw.githubusercontent.com/fchollet/keras/master/examples/lstm_text_generation.py" rel="nofollow">https://raw.githubusercontent.com/fchollet/keras/master/examples/lstm_text_generation.py</a></p>
  
  <p>It works on quite a big demo text set from Nietzsche and says</p>
  
  <blockquote>
    <p>If you try this script on new data, make sure your corpus
    has at least ~100k characters. ~1M is better.</p>
  </blockquote>
  
  <p>This pops up a couple of questions.</p>
  
  <p>A.) If all I want is an AI with a very limited vocabulary where the generate text should be short sentences following a basic pattern.</p>
  
  <p>E.g.</p>
  
  <p><em>I like blue sky with white clouds</em></p>
  
  <p><em>I like yellow fields with some trees</em></p>
  
  <p><em>I like big cities with lots of bars</em></p>
  
  <p>...</p>
  
  <p>Would it then be reasonable to use a much much smaller dataset?</p>
  
  <p>B.) If the dataset really needs to be that big. What if I just repeat the text over and over to reach the recommended minimum? If that would work though, I'd be wondering how that is any different from just taking more iterations of learning with the same shorter text?</p>
  
  <p>Obviously I can play with these two questions myself and in fact I am experimenting with it. One thing I already figured out is that with a shorter text following a basic pattern I can get to a very very low ( ~0.04) quite fast but the predicted text just turns out as gibberish.</p>
  
  <p>My naive explanation for that would be that there are just not enough samples to proof against whether the gibberish actually makes sense or not? But then again I wonder if more iterations or duplicating the content would actually help.</p>
  
  <p>I'm trying to experiment with these questions myself so please don't think I'm just too lazy and are aiming for others to do the work. I'm just looking for more experienced people to give me a better understanding of the mechanics that influence these things.</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-08-09T07:04:07.530Z
Comments.Text: If the patterns are as simple as you describe, you could always try *generating* a corpus.
Comments.UserDisplayName: NietzscheanAI
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-08-09T12:50:04.383Z
Comments.Text: You mean, generate a corpus for the learning? Yep, I could generate a corpus where I repeat lots of sentences. I just wonder if that's comparable to just use a smaller dataset (without repeated sentences) and iterate on it more often.
Comments.UserDisplayName: Christoph
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-08-09T12:53:20.937Z
Comments.Text: No, I mean: define a grammar template of the form "I like A B with C D" (or similar) and use something like WordNet/ConceptNet to help you generate lots of instances of the template.
Comments.UserDisplayName: NietzscheanAI
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-08-11T07:15:18.197Z
Comments.Text: Got it. But I'm not so much interested in solving the "lack of data" problem. Rather I'd like to understand why the NN doesn't learn to write (albeit very basic) sentences if it sees a handful of basic sentences over and over. I tried that script with 5 different sentences repeated to a char count of > 1M and still after ~40 iterations it's gibberish. I'm now testing out the original one that's based on a Nietzsche text to see the difference.
Comments.UserDisplayName: Christoph
Tags: <datasets><long-short-term-memory>
Title: In LSTM text generation can low amount of training data be compensated?
UserDisplayName: Christoph
ThreadText: # In LSTM text generation can low amount of training data be compensated?
  
  (Asked by: username_0 on 2016-08-09)
  
  I'm playing with an LSTM to generate text. In particular, this one:
  
  <https://raw.githubusercontent.com/fchollet/keras/master/examples/lstm_text_generation.py>
  
  It works on quite a big demo text set from Nietzsche and says
  
  > If you try this script on new data, make sure your corpus
  > has at least ~100k characters. ~1M is better.
  
  This pops up a couple of questions.
  
  A.) If all I want is an AI with a very limited vocabulary where the generate text should be short sentences following a basic pattern.
  
  E.g.
  
  *I like blue sky with white clouds*
  
  *I like yellow fields with some trees*
  
  *I like big cities with lots of bars*
  
  ...
  
  Would it then be reasonable to use a much much smaller dataset?
  
  B.) If the dataset really needs to be that big. What if I just repeat the text over and over to reach the recommended minimum? If that would work though, I'd be wondering how that is any different from just taking more iterations of learning with the same shorter text?
  
  Obviously I can play with these two questions myself and in fact I am experimenting with it. One thing I already figured out is that with a shorter text following a basic pattern I can get to a very very low ( ~0.04) quite fast but the predicted text just turns out as gibberish.
  
  My naive explanation for that would be that there are just not enough samples to proof against whether the gibberish actually makes sense or not? But then again I wonder if more iterations or duplicating the content would actually help.
  
  I'm trying to experiment with these questions myself so please don't think I'm just too lazy and are aiming for others to do the work. I'm just looking for more experienced people to give me a better understanding of the mechanics that influence these things.
  
  **Tags:** datasets, long-short-term-memory
  > **Comments:**
  > **username_1**: If the patterns are as simple as you describe, you could always try *generating* a corpus.  
  > **username_0**: You mean, generate a corpus for the learning? Yep, I could generate a corpus where I repeat lots of sentences. I just wonder if that's comparable to just use a smaller dataset (without repeated sentences) and iterate on it more often.  
  > **username_1**: No, I mean: define a grammar template of the form "I like A B with C D" (or similar) and use something like WordNet/ConceptNet to help you generate lots of instances of the template.  
  > **username_0**: Got it. But I'm not so much interested in solving the "lack of data" problem. Rather I'd like to understand why the NN doesn't learn to write (albeit very basic) sentences if it sees a handful of basic sentences over and over. I tried that script with 5 different sentences repeated to a char count of > 1M and still after ~40 iterations it's gibberish. I'm now testing out the original one that's based on a Nietzsche text to see the difference.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2017-04-24. Score: 2
  
  A.) The algorithm can only learn patterns from the data, so if you want it to align to a specific functional form of language, it would make sense to curate your data to only contain sentences of that very structure.
  
  If you tune your hyper-parameters such that the model is not as complex as the default, it should work for datasets that are smaller. But this relies on the observation that the thing you are trying to learn isn't as complex (as opposed to learning from a large corpus of Nietzsche).
  
  B.) What is happening is that you are *overfitting the data*, such that the LSTM isn't generalizing to your intended goal. In essence, overfitting means that your model is learning irrelevant details that *by chance* happen to predict the intended goal in the training data. To alleviate this, use a validation set.
  
  And finally (but very importantly), these machine learning techniques don't learn the same way as humans do. To understand why you'd need to learn the math behind NNs, and I can't think of an intuitive explanation as to why.
  
  
  ---
  
  ### Answer by username_3 on 2017-04-25. Score: 0
  
  LSTMs are good and everything but you can try some exotic methods like Convolutional Neural Network for NLP. Perhaps it would be more suited for this need. Have a look at this.
  
  1. <http://www.wildml.com/2016/01/attention-and-memory-in-deep-learning-and-nlp/>
  2. <http://www.wildml.com/2015/12/implementing-a-cnn-for-text-classification-in-tensorflow/>
  
  
  ---
  
  