Answers.Body: <p>The current state of the art in natural language generation are all auto-regressive transformer models. Transformers no longer use recurrent neural networks such as LSTM, because the recurrences makes long dependencies messy to calculate. Instead, Transformers only keep the attention layers, and apply attention on all the existing text so far, which can be done in parallel so therefore very fast, while being able to attend to long dependencies (e.g. understanding that &quot;it&quot; refers to &quot;John&quot; from 3 sentences ago). They are also faster to train than LSTMs (on powerful GPUs at least). The downside is more memory requirement, and you need large models and large datasets (LSTMs work better for small models and small datasets). <a href="http://jalammar.github.io/illustrated-transformer/" rel="nofollow noreferrer">Here</a> is some background info on how they work.</p>
  <p>Auto-regressive transformer models only use the decoder for text generation, and removes the encoder. Given an input, they predict the next word.</p>
  <p>The most well-known one is GPT (<a href="https://github.com/openai/gpt-3" rel="nofollow noreferrer">GPT-3</a> has 175B parameters; <a href="https://github.com/openai/gpt-2" rel="nofollow noreferrer">GPT-2</a> has 1.5B parameters, and GPT-1 has 175M parameters) GPT is developed by OpenAI and is a commercial, paid-software if you want to use their official model, but I'm sure with a little digging you can find <a href="https://huggingface.co/transformers/pretrained_models.html" rel="nofollow noreferrer">community-trained models</a> that will perform slightly worse but is at least free to use. GPT is basically a vanilla transformer, but trained on a huge, huge dataset with a huge, huge model to achieve state-of-the-art performance.</p>
  <p>Other auto-regressive transformer models include:</p>
  <ul>
  <li><a href="http://CTRL" rel="nofollow noreferrer">CTRL</a> by Salesforce, which uses the novel idea of control codes to guide the style of generation (e.g. to generate Wikipedia article style text or book review style text).</li>
  <li><a href="https://github.com/zihangdai/xlnet" rel="nofollow noreferrer">XLNet</a> by Google AI Brain team, which handles longer sequences more accuractely than the others because it re-introduces reccurrence back in the transformer model, allowing it to remember past sequences. Otherwise, vanilla transformers cannot handle dependencies that crosses sequences (note: a sequence is limited by the max length you can feed into the model, bottle-necked by your memory requirement, and can contain many sentences or paragraphs).</li>
  <li><a href="https://github.com/lucidrains/reformer-pytorch" rel="nofollow noreferrer">Reformer</a> by Google Research, which is a more efficient transformer that significantly reduces the memory requirement while also being faster to compute on long sequences.</li>
  </ul>
  <p>If your goal is just to generate English or another commonly researched language, you can use an existing pre-trained language model and avoid doing any training yourself. This saves a lot of time, and there should at least be free community-trained models readily available. Otherwise, for obscure tasks, you'd have to train one yourself, and these state of the art models will take immense resources.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-18T20:50:22.137Z
Answers.LastActivityDate: 2020-12-18T21:25:12.747Z
Answers.LastEditDate: 2020-12-18T21:25:12.747Z
Answers.UserDisplayName: user3667125
Body: <p>I have a trained LSTM language model and want to use it to generate text. The standard approach for this seems to be:</p>
  <ol>
  <li>Apply softmax function</li>
  <li>Take a weighted random choice to determine the next word</li>
  </ol>
  <p>This is working reasonably well for me, but it would be nice to play around with other options. Are there any good alternatives to this?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-18T18:21:55.910Z
Comments.Text: Sometimes, [expert system](https://en.wikipedia.org/wiki/Expert_system) techniques might be used to generate text, e.g. with [CLIPS](http://www.clipsrules.net/). You might then be interested by [RefPerSys](http://refpersys.org/). BTW, I guess that different techniques apply to generate Chinese, Russian, or French, and things are different if you want to generate audio files or [UTF-8](https://utf8everywhere.org/) textual files, or [PDF](https://en.wikipedia.org/wiki/PDF) documents
Comments.UserDisplayName: Basile Starynkevitch
Tags: <natural-language-processing><reference-request><language-model><text-generation>
Title: Are there any good alternatives to an LSTM language model for text generation?
UserDisplayName: Christian Doucette
ThreadText: # Are there any good alternatives to an LSTM language model for text generation?
  
  (Asked by: username_0 on 2020-12-18)
  
  I have a trained LSTM language model and want to use it to generate text. The standard approach for this seems to be:
  
  1. Apply softmax function
  2. Take a weighted random choice to determine the next word
  
  This is working reasonably well for me, but it would be nice to play around with other options. Are there any good alternatives to this?
  
  **Tags:** natural-language-processing, reference-request, language-model, text-generation
  > **Comments:**
  > **username_1**: Sometimes, [expert system](https://en.wikipedia.org/wiki/Expert_system) techniques might be used to generate text, e.g. with [CLIPS](http://www.clipsrules.net/). You might then be interested by [RefPerSys](http://refpersys.org/). BTW, I guess that different techniques apply to generate Chinese, Russian, or French, and things are different if you want to generate audio files or [UTF-8](https://utf8everywhere.org/) textual files, or [PDF](https://en.wikipedia.org/wiki/PDF) documents  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-12-18. Score: 3 | Accepted answer
  
  The current state of the art in natural language generation are all auto-regressive transformer models. Transformers no longer use recurrent neural networks such as LSTM, because the recurrences makes long dependencies messy to calculate. Instead, Transformers only keep the attention layers, and apply attention on all the existing text so far, which can be done in parallel so therefore very fast, while being able to attend to long dependencies (e.g. understanding that "it" refers to "John" from 3 sentences ago). They are also faster to train than LSTMs (on powerful GPUs at least). The downside is more memory requirement, and you need large models and large datasets (LSTMs work better for small models and small datasets). [Here](http://jalammar.github.io/illustrated-transformer/) is some background info on how they work.
  
  Auto-regressive transformer models only use the decoder for text generation, and removes the encoder. Given an input, they predict the next word.
  
  The most well-known one is GPT ([GPT-3](https://github.com/openai/gpt-3) has 175B parameters; [GPT-2](https://github.com/openai/gpt-2) has 1.5B parameters, and GPT-1 has 175M parameters) GPT is developed by OpenAI and is a commercial, paid-software if you want to use their official model, but I'm sure with a little digging you can find [community-trained models](https://huggingface.co/transformers/pretrained_models.html) that will perform slightly worse but is at least free to use. GPT is basically a vanilla transformer, but trained on a huge, huge dataset with a huge, huge model to achieve state-of-the-art performance.
  
  Other auto-regressive transformer models include:
  
  * [CTRL](http://CTRL) by Salesforce, which uses the novel idea of control codes to guide the style of generation (e.g. to generate Wikipedia article style text or book review style text).
  * [XLNet](https://github.com/zihangdai/xlnet) by Google AI Brain team, which handles longer sequences more accuractely than the others because it re-introduces reccurrence back in the transformer model, allowing it to remember past sequences. Otherwise, vanilla transformers cannot handle dependencies that crosses sequences (note: a sequence is limited by the max length you can feed into the model, bottle-necked by your memory requirement, and can contain many sentences or paragraphs).
  * [Reformer](https://github.com/lucidrains/reformer-pytorch) by Google Research, which is a more efficient transformer that significantly reduces the memory requirement while also being faster to compute on long sequences.
  
  If your goal is just to generate English or another commonly researched language, you can use an existing pre-trained language model and avoid doing any training yourself. This saves a lot of time, and there should at least be free community-trained models readily available. Otherwise, for obscure tasks, you'd have to train one yourself, and these state of the art models will take immense resources.
  
  
  ---
  
  