Answers.Body: <p>A few reasons:</p>
  <ol>
  <li>Transformers are amazing at text generation already (e.g. <a href="https://arxiv.org/abs/2005.14165" rel="nofollow noreferrer">GPT-3</a> which <a href="https://lacker.io/ai/2020/07/06/giving-gpt-3-a-turing-test.html" rel="nofollow noreferrer">almost passes</a> the <a href="https://en.wikipedia.org/wiki/Turing_test" rel="nofollow noreferrer">Turing-Test</a>)</li>
  <li><a href="https://www.quora.com/Can-one-use-generative-adversarial-networks-in-NLP-related-problems" rel="nofollow noreferrer">The original GAN requires a continuous data representation</a> (e.g. images) instead of a discrete one (e.g. text), so that slight error signals can be used for learning.</li>
  <li>Empirically speaking, GANs don't seem to work that well on non-image data. I recently applied them to regular tabular data but found auto-encoders much more useful.</li>
  </ol>
  <p>P.S. Also it's my intuition that GAN's and <a href="https://proceedings.neurips.cc/paper/2020/file/4c5bcfec8584af0d967f1ab10179ca4b-Paper.pdf" rel="nofollow noreferrer">Denoising Diffusion Models</a> are popular in the <em>image generation community</em> because they both offer mechanisms that can <a href="https://ai.stackexchange.com/questions/8885/why-is-the-variational-auto-encoders-output-blurred-while-gans-output-is-crisp"><strong>prevent &quot;blurriness&quot;</strong></a> in the generated images (either the adversary or extra denoising iterations).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-25T10:36:52.210Z
Answers.Text: Please, remove the very arguable sentence "which essentially passes the Turing-Test". Alternatively, can you provide some real evidence for it? If it generates some meaningful text it doesn't mean passed the Turing test. I'm sure it doesn't pass the imitation game.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-28T18:17:43.513Z
Answers.Text: I edited it to include a reference, and say 'almost passes' since it still isn't great at answering nonsense questions.
Answers.UserDisplayName: profPlum
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-28T18:18:37.207Z
Answers.Text: I should mention however that when GPT-3 is primed with examples (via meta-learning) of calling out nonsense it does learn the skill immediately: https://arr.am/2020/07/25/gpt-3-uncertainty-prompts/
Answers.UserDisplayName: profPlum
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-24T18:24:31.867Z
Answers.LastActivityDate: 2024-01-23T17:45:22.113Z
Answers.LastEditDate: 2024-01-23T17:45:22.113Z
Answers.UserDisplayName: profPlum
Body: <p>I am wondering why there has not been more usage of GANs for NLP. I know there has been research on the subject (The Google Scholar page for the subject is <a href="https://scholar.google.com/scholar?q=GANs+for+nlp&amp;hl=en&amp;as_sdt=0&amp;as_vis=1&amp;oi=scholart" rel="nofollow noreferrer">here</a>).</p>
  <p>Are there any specific reasons why GANs do not work for NLP specifically VQGAN + CLIP variants? I do not understand why most text generated by AI is done through predicting the next letter or word in a sequence with RNNs when GANs have had so much success generating deep fakes and the such instead of say, predicting the next pixel.</p>
  
Tags: <natural-language-processing><generative-adversarial-networks>
Title: Why are GAN models not heavily used for NLP?
UserDisplayName: catasaurus
ThreadText: # Why are GAN models not heavily used for NLP?
  
  (Asked by: username_0 on 2022-03-24)
  
  I am wondering why there has not been more usage of GANs for NLP. I know there has been research on the subject (The Google Scholar page for the subject is [here](https://scholar.google.com/scholar?q=GANs+for+nlp&hl=en&as_sdt=0&as_vis=1&oi=scholart)).
  
  Are there any specific reasons why GANs do not work for NLP specifically VQGAN + CLIP variants? I do not understand why most text generated by AI is done through predicting the next letter or word in a sequence with RNNs when GANs have had so much success generating deep fakes and the such instead of say, predicting the next pixel.
  
  **Tags:** natural-language-processing, generative-adversarial-networks
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-03-24. Score: 6 | Accepted answer
  
  A few reasons:
  
  1. Transformers are amazing at text generation already (e.g. [GPT-3](https://arxiv.org/abs/2005.14165) which [almost passes](https://lacker.io/ai/2020/07/06/giving-gpt-3-a-turing-test.html) the [Turing-Test](https://en.wikipedia.org/wiki/Turing_test))
  2. [The original GAN requires a continuous data representation](https://www.quora.com/Can-one-use-generative-adversarial-networks-in-NLP-related-problems) (e.g. images) instead of a discrete one (e.g. text), so that slight error signals can be used for learning.
  3. Empirically speaking, GANs don't seem to work that well on non-image data. I recently applied them to regular tabular data but found auto-encoders much more useful.
  
  P.S. Also it's my intuition that GAN's and [Denoising Diffusion Models](https://proceedings.neurips.cc/paper/2020/file/4c5bcfec8584af0d967f1ab10179ca4b-Paper.pdf) are popular in the *image generation community* because they both offer mechanisms that can [**prevent "blurriness"**](https://ai.stackexchange.com/questions/8885/why-is-the-variational-auto-encoders-output-blurred-while-gans-output-is-crisp) in the generated images (either the adversary or extra denoising iterations).
  
  > **Comments:**
  > **username_2**: Please, remove the very arguable sentence "which essentially passes the Turing-Test". Alternatively, can you provide some real evidence for it? If it generates some meaningful text it doesn't mean passed the Turing test. I'm sure it doesn't pass the imitation game.  
  > **username_1**: I edited it to include a reference, and say 'almost passes' since it still isn't great at answering nonsense questions.  
  > **username_1**: I should mention however that when GPT-3 is primed with examples (via meta-learning) of calling out nonsense it does learn the skill immediately: https://arr.am/2020/07/25/gpt-3-uncertainty-prompts/  
  
  ---
  
  