<p>A few reasons:</p>
<ol>
<li>Transformers are amazing at text generation already (e.g. <a href="https://arxiv.org/abs/2005.14165" rel="nofollow noreferrer">GPT-3</a> which <a href="https://lacker.io/ai/2020/07/06/giving-gpt-3-a-turing-test.html" rel="nofollow noreferrer">almost passes</a> the <a href="https://en.wikipedia.org/wiki/Turing_test" rel="nofollow noreferrer">Turing-Test</a>)</li>
<li><a href="https://www.quora.com/Can-one-use-generative-adversarial-networks-in-NLP-related-problems" rel="nofollow noreferrer">The original GAN requires a continuous data representation</a> (e.g. images) instead of a discrete one (e.g. text), so that slight error signals can be used for learning.</li>
<li>Empirically speaking, GANs don't seem to work that well on non-image data. I recently applied them to regular tabular data but found auto-encoders much more useful.</li>
</ol>
<p>P.S. Also it's my intuition that GAN's and <a href="https://proceedings.neurips.cc/paper/2020/file/4c5bcfec8584af0d967f1ab10179ca4b-Paper.pdf" rel="nofollow noreferrer">Denoising Diffusion Models</a> are popular in the <em>image generation community</em> because they both offer mechanisms that can <a href="https://ai.stackexchange.com/questions/8885/why-is-the-variational-auto-encoders-output-blurred-while-gans-output-is-crisp"><strong>prevent &quot;blurriness&quot;</strong></a> in the generated images (either the adversary or extra denoising iterations).</p>

CC BY-SA 4.0
2022-03-25T10:36:52.210Z
Please, remove the very arguable sentence "which essentially passes the Turing-Test". Alternatively, can you provide some real evidence for it? If it generates some meaningful text it doesn't mean passed the Turing test. I'm sure it doesn't pass the imitation game.
nbro
CC BY-SA 4.0
2022-03-28T18:17:43.513Z
I edited it to include a reference, and say 'almost passes' since it still isn't great at answering nonsense questions.
profPlum
CC BY-SA 4.0
2022-03-28T18:18:37.207Z
I should mention however that when GPT-3 is primed with examples (via meta-learning) of calling out nonsense it does learn the skill immediately: https://arr.am/2020/07/25/gpt-3-uncertainty-prompts/
profPlum
CC BY-SA 4.0
2022-03-24T18:24:31.867Z
2024-01-23T17:45:22.113Z
2024-01-23T17:45:22.113Z
profPlum
<p>I am wondering why there has not been more usage of GANs for NLP. I know there has been research on the subject (The Google Scholar page for the subject is <a href="https://scholar.google.com/scholar?q=GANs+for+nlp&amp;hl=en&amp;as_sdt=0&amp;as_vis=1&amp;oi=scholart" rel="nofollow noreferrer">here</a>).</p>
<p>Are there any specific reasons why GANs do not work for NLP specifically VQGAN + CLIP variants? I do not understand why most text generated by AI is done through predicting the next letter or word in a sequence with RNNs when GANs have had so much success generating deep fakes and the such instead of say, predicting the next pixel.</p>

<natural-language-processing><generative-adversarial-networks>
Why are GAN models not heavily used for NLP?
catasaurus
# Why are GAN models not heavily used for NLP?

(Asked by: username_0 on 2022-03-24)

I am wondering why there has not been more usage of GANs for NLP. I know there has been research on the subject (The Google Scholar page for the subject is [here](https://scholar.google.com/scholar?q=GANs+for+nlp&hl=en&as_sdt=0&as_vis=1&oi=scholart)).

Are there any specific reasons why GANs do not work for NLP specifically VQGAN + CLIP variants? I do not understand why most text generated by AI is done through predicting the next letter or word in a sequence with RNNs when GANs have had so much success generating deep fakes and the such instead of say, predicting the next pixel.

**Tags:** natural-language-processing, generative-adversarial-networks


---

## 1 answers

### Answer by username_1 on 2022-03-24. Score: 6 | Accepted answer

A few reasons:

1. Transformers are amazing at text generation already (e.g. [GPT-3](https://arxiv.org/abs/2005.14165) which [almost passes](https://lacker.io/ai/2020/07/06/giving-gpt-3-a-turing-test.html) the [Turing-Test](https://en.wikipedia.org/wiki/Turing_test))
2. [The original GAN requires a continuous data representation](https://www.quora.com/Can-one-use-generative-adversarial-networks-in-NLP-related-problems) (e.g. images) instead of a discrete one (e.g. text), so that slight error signals can be used for learning.
3. Empirically speaking, GANs don't seem to work that well on non-image data. I recently applied them to regular tabular data but found auto-encoders much more useful.

P.S. Also it's my intuition that GAN's and [Denoising Diffusion Models](https://proceedings.neurips.cc/paper/2020/file/4c5bcfec8584af0d967f1ab10179ca4b-Paper.pdf) are popular in the *image generation community* because they both offer mechanisms that can [**prevent "blurriness"**](https://ai.stackexchange.com/questions/8885/why-is-the-variational-auto-encoders-output-blurred-while-gans-output-is-crisp) in the generated images (either the adversary or extra denoising iterations).

> **Comments:**
> **username_2**: Please, remove the very arguable sentence "which essentially passes the Turing-Test". Alternatively, can you provide some real evidence for it? If it generates some meaningful text it doesn't mean passed the Turing test. I'm sure it doesn't pass the imitation game.  
> **username_1**: I edited it to include a reference, and say 'almost passes' since it still isn't great at answering nonsense questions.  
> **username_1**: I should mention however that when GPT-3 is primed with examples (via meta-learning) of calling out nonsense it does learn the skill immediately: https://arr.am/2020/07/25/gpt-3-uncertainty-prompts/  

---

