Answers.Body: <p>Recurrent Neural Networks (RNNs) have been applied to generate text. In <a href="https://karpathy.github.io/2015/05/21/rnn-effectiveness/" rel="nofollow noreferrer">this</a> blog post you will find a couple of interesting text examples (the author also has made his code available on github), e.g. their Shakespeare-like texts generated by an RNN:</p>
  
  <blockquote>
    <p>PANDARUS:
    Alas, I think he shall be come approached and the day
    When little srain would be attain'd into being never fed,
    And who is but a chain and subjects of his death,
    I should not sleep.</p>
    
    <p>Second Senator:
    They are away this miseries, produced upon my soul,
    Breaking and strongly should be buried, when I perish
    The earth and thoughts of many states.</p>
    
    <p>DUKE VINCENTIO:
    Well, your wit is in the care of side and that.</p>
    
    <p>Second Lord:
    They would be ruled after this chamber, and
    my fair nues begun out of the fact, to be conveyed,
    Whose noble souls I'll have the heart of the wars.</p>
    
    <p>Clown:
    Come, sir, I will make did behold your worship.</p>
    
    <p>VIOLA:
    I'll drink it.</p>
  </blockquote>
  
  <p>As you can see the RNN is able to somewhat mimic the "flow" of the texts it has been trained on but some sentences (like at the very end) do not make much contextual sense. </p>
  
  <p>Moreover, RNNs have been trained to generate other content, e.g. drawing numbers (see <a href="https://arxiv.org/pdf/1502.04623.pdf" rel="nofollow noreferrer">here</a>) or creating music (see <a href="http://people.idsia.ch/~juergen/blues/IDSIA-07-02.pdf" rel="nofollow noreferrer">here</a>).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-07T15:33:39.007Z
Answers.Text: is it a good approach to train the RNN on a large amounts of data (sentences, paragraphs...) ?
Answers.UserDisplayName: Thorvald
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-07T15:44:31.710Z
Answers.Text: Well, that depends on a number of things: what you define as "large amount", your computing power, the amount of data needed to learn the respective task etc.  In the linked blog post the author used up to 100MB to train his Wikipedia article writing RNN. Using a rough ball park figure 100MB of pure text equals about 100 books with 500 pages each.
Answers.UserDisplayName: Jonathan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-07T16:07:33.613Z
Answers.Text: okay, I got you! so the best approach here is using RNNs and go there! Thank you for your time!
Answers.UserDisplayName: Thorvald
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-07T15:27:59.587Z
Answers.LastActivityDate: 2019-12-07T15:35:41.067Z
Answers.LastEditDate: 2019-12-07T15:35:41.067Z
Answers.UserDisplayName: Jonathan
Answers.Body: <p>There have been many methods proposed for text generating, but recurrent network dominates natural language processing with a key component: the perception of time.</p>
  <p>Many networks have been tried for text generation, with notable examples such as Markov chain. However RNN have been proven to work the best and is dominating the field of language modelling (text generation).</p>
  <h1>How text generation works</h1>
  <p>A neural network that generates text is commonly called a language model. It is trained on large amount of text with labels being the next token. The text generation process uses several random token as the starting phrase and then the network predicts the rest. However the network does not just predicts the most probable word, instead it randomly chooses one of the few most probable token, hence the generating part.</p>
  <h1>Why RNN work best on language modelling</h1>
  <p>RNN have a perception of time built into the architecture of teh network. LSTM, a popular RNN variant used, is composed of &quot;memory units&quot; that &quot;remembers&quot; past text, thus the &quot;time&quot; part. RNN process input according to the sequence of time, so the network can naturally understand time, thus the superior performance compared to other networks.</p>
  <h1>Architecture of language model</h1>
  <p>A language model consists of the encoder and the decoder. The encoder compresses word one-hot representation to a smaller sized vector representation. The smaller sized representation is then passed through the decoder, which maps the encoding to the words one hot vectors again.</p>
  <h1>State of the art results for language modelling</h1>
  <p>Language modelling is an actively researched field in the AI community, and recently the model GPT-2 have achieved a breakthrough in language modelling accuracy, producing almost human like text with a special component added, the attention layer. Attention basically maps the &quot;memory states&quot; of the encoder and feed it as input to the decoder. The data teh model is trained on is also very large, with over 20GB of web scraped data from sites like Reddit.</p>
  <h1>Limits of language modelling</h1>
  <p>One limit of language modelling is the size of generated text. As GPU don't have unlimited memory, language model usually limits the input token size to a specific number, padding or trimming to this number. The number is usually 500-1000, which includes a paragraph or two, but not an entire book. You can only generate short paragraphs and essay with language modelling. For long text it is much harder.</p>
  <h1>Resources to help you get started</h1>
  <p>GPT-2 open AI blog: <a href="https://openai.com/blog/better-language-models/" rel="nofollow noreferrer">https://openai.com/blog/better-language-models/</a>
  GPT-2 online interactive site for text generation: <a href="https://talktotransformer.com/" rel="nofollow noreferrer">https://talktotransformer.com/</a>
  How to train and fine tune GPT-2 in python: <a href="https://minimaxir.com/2019/09/howto-gpt2/" rel="nofollow noreferrer">https://minimaxir.com/2019/09/howto-gpt2/</a></p>
  <p>Hope I can help you</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-07T19:13:08.360Z
Answers.Text: So what to do when I'm trying to generate large texts that for sure will exceeds 1000 words? should I do it piece by piece?
Answers.UserDisplayName: Thorvald
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-08T05:27:40.883Z
Answers.Text: In short, you simply cannot. The training data of the model is just not a piece of continuous long text, so at the end the AI is going to predict the end token and the sentence/paragraph is going to end.
Answers.UserDisplayName: Clement
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-07T17:06:04.333Z
Answers.LastActivityDate: 2019-12-07T17:06:04.333Z
Answers.LastEditDate: 2020-06-17T09:57:20.710Z
Answers.UserDisplayName: Clement
Body: <p>Now I know this might break some StackExchange rules and I am definitely open for taking the thread down if it does!
  I am trying to build an AI that can write it's own book and I have no idea where to start or what are the appropriate algorithms and approaches to go with. 
  How should I start and what do I exactly need for such a project? </p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-07T15:05:18.340Z
Comments.Text: Text generation has a long history in computing, and there are plenty of AI areas that might help. This question might be on topic if you explain more about your project goals and where you are stuck. I thnk some details on the nature of the book and its purpose are important, because state of the art language algorithms can generate text with certain stylistic forms, but which is usually vacuous nonsense. Take a look at https://talktotransformer.com/ perhaps - is that the sort of text generation you had in mind?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-07T15:14:28.337Z
Comments.Text: something like that but not quite exactly, I want to do something like feed the AI some history books about a country (this is just an example because I can't disclose the purpose of the AI) and it generates an alternate history for that country
Comments.UserDisplayName: Thorvald
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-07T15:23:17.000Z
Comments.Text: OK, make up some alternative but similar purpose like the one you suggest and add it to your question. I think that what you want is beyond current AI techniques, especially if you expect anything the AI produces to many any kind of *sense*. So it may help to understand whether your purpose is in the question to really make such an AI (an impossible task for 2020) or to better understand the technologies that could be involved in such a project and why it is hard to do
Comments.UserDisplayName: Neil Slater
Tags: <neural-networks><machine-learning><deep-learning><artificial-neuron>
Title: Building an AI that generates text by itself
UserDisplayName: Thorvald
ThreadText: # Building an AI that generates text by itself
  
  (Asked by: username_0 on 2019-12-07)
  
  Now I know this might break some StackExchange rules and I am definitely open for taking the thread down if it does!
  I am trying to build an AI that can write it's own book and I have no idea where to start or what are the appropriate algorithms and approaches to go with.
  How should I start and what do I exactly need for such a project?
  
  **Tags:** neural-networks, machine-learning, deep-learning, artificial-neuron
  
  > **Comments:**
  > **username_1**: Text generation has a long history in computing, and there are plenty of AI areas that might help. This question might be on topic if you explain more about your project goals and where you are stuck. I thnk some details on the nature of the book and its purpose are important, because state of the art language algorithms can generate text with certain stylistic forms, but which is usually vacuous nonsense. Take a look at https://talktotransformer.com/ perhaps - is that the sort of text generation you had in mind?  
  > **username_0**: something like that but not quite exactly, I want to do something like feed the AI some history books about a country (this is just an example because I can't disclose the purpose of the AI) and it generates an alternate history for that country  
  > **username_1**: OK, make up some alternative but similar purpose like the one you suggest and add it to your question. I think that what you want is beyond current AI techniques, especially if you expect anything the AI produces to many any kind of *sense*. So it may help to understand whether your purpose is in the question to really make such an AI (an impossible task for 2020) or to better understand the technologies that could be involved in such a project and why it is hard to do  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2019-12-07. Score: 3 | Accepted answer
  
  There have been many methods proposed for text generating, but recurrent network dominates natural language processing with a key component: the perception of time.
  
  Many networks have been tried for text generation, with notable examples such as Markov chain. However RNN have been proven to work the best and is dominating the field of language modelling (text generation).
  
  # How text generation works
  
  A neural network that generates text is commonly called a language model. It is trained on large amount of text with labels being the next token. The text generation process uses several random token as the starting phrase and then the network predicts the rest. However the network does not just predicts the most probable word, instead it randomly chooses one of the few most probable token, hence the generating part.
  
  # Why RNN work best on language modelling
  
  RNN have a perception of time built into the architecture of teh network. LSTM, a popular RNN variant used, is composed of "memory units" that "remembers" past text, thus the "time" part. RNN process input according to the sequence of time, so the network can naturally understand time, thus the superior performance compared to other networks.
  
  # Architecture of language model
  
  A language model consists of the encoder and the decoder. The encoder compresses word one-hot representation to a smaller sized vector representation. The smaller sized representation is then passed through the decoder, which maps the encoding to the words one hot vectors again.
  
  # State of the art results for language modelling
  
  Language modelling is an actively researched field in the AI community, and recently the model GPT-2 have achieved a breakthrough in language modelling accuracy, producing almost human like text with a special component added, the attention layer. Attention basically maps the "memory states" of the encoder and feed it as input to the decoder. The data teh model is trained on is also very large, with over 20GB of web scraped data from sites like Reddit.
  
  # Limits of language modelling
  
  One limit of language modelling is the size of generated text. As GPU don't have unlimited memory, language model usually limits the input token size to a specific number, padding or trimming to this number. The number is usually 500-1000, which includes a paragraph or two, but not an entire book. You can only generate short paragraphs and essay with language modelling. For long text it is much harder.
  
  # Resources to help you get started
  
  GPT-2 open AI blog: <https://openai.com/blog/better-language-models/>
  GPT-2 online interactive site for text generation: <https://talktotransformer.com/>
  How to train and fine tune GPT-2 in python: <https://minimaxir.com/2019/09/howto-gpt2/>
  
  Hope I can help you
  
  > **Comments:**
  > **username_0**: So what to do when I'm trying to generate large texts that for sure will exceeds 1000 words? should I do it piece by piece?  
  > **username_3**: In short, you simply cannot. The training data of the model is just not a piece of continuous long text, so at the end the AI is going to predict the end token and the sentence/paragraph is going to end.  
  
  ---
  
  ### Answer by username_2 on 2019-12-07. Score: 2
  
  Recurrent Neural Networks (RNNs) have been applied to generate text. In [this](https://karpathy.github.io/2015/05/21/rnn-effectiveness/) blog post you will find a couple of interesting text examples (the author also has made his code available on github), e.g. their Shakespeare-like texts generated by an RNN:
  
  > PANDARUS:
  > Alas, I think he shall be come approached and the day
  > When little srain would be attain'd into being never fed,
  > And who is but a chain and subjects of his death,
  > I should not sleep.
  >
  > Second Senator:
  > They are away this miseries, produced upon my soul,
  > Breaking and strongly should be buried, when I perish
  > The earth and thoughts of many states.
  >
  > DUKE VINCENTIO:
  > Well, your wit is in the care of side and that.
  >
  > Second Lord:
  > They would be ruled after this chamber, and
  > my fair nues begun out of the fact, to be conveyed,
  > Whose noble souls I'll have the heart of the wars.
  >
  > Clown:
  > Come, sir, I will make did behold your worship.
  >
  > VIOLA:
  > I'll drink it.
  
  As you can see the RNN is able to somewhat mimic the "flow" of the texts it has been trained on but some sentences (like at the very end) do not make much contextual sense.
  
  Moreover, RNNs have been trained to generate other content, e.g. drawing numbers (see [here](https://arxiv.org/pdf/1502.04623.pdf)) or creating music (see [here](http://people.idsia.ch/~juergen/blues/IDSIA-07-02.pdf)).
  
  > **Comments:**
  > **username_0**: is it a good approach to train the RNN on a large amounts of data (sentences, paragraphs...) ?  
  > **username_2**: Well, that depends on a number of things: what you define as "large amount", your computing power, the amount of data needed to learn the respective task etc.  In the linked blog post the author used up to 100MB to train his Wikipedia article writing RNN. Using a rough ball park figure 100MB of pure text equals about 100 books with 500 pages each.  
  > **username_0**: okay, I got you! so the best approach here is using RNNs and go there! Thank you for your time!  
  
  ---
  
  