Answers.Body: <p>When you sample a sentence using an RNN, at each step <span class="math-container">$t$</span>, if <span class="math-container">$N$</span> is the size of your vocabulary, <span class="math-container">$W$</span> the array with the <span class="math-container">$N$</span> words in your vocabulary, and <span class="math-container">$f$</span> your RNN without its final linear layer, you do:</p>
  <p><span class="math-container">$$p_{t+1} = \text{softmax}(f(w_t))$$</span>
  <span class="math-container">$$i_{t+1}  = \text{argmax}(p_{t+1})$$</span>
  <span class="math-container">$$w_{t+1} = W[i_{t+1}]$$</span>
  where <span class="math-container">$p_{t+1}$</span> is a vector of probabilities of size <span class="math-container">$N$</span>, and <span class="math-container">$w_{t+1}$</span> a word from the vocabulary.</p>
  <p>The <span class="math-container">$\text{argmax}$</span> operation to get <span class="math-container">$w_{t+1}$</span> is not differentiable.</p>
  <p>I'm not sure what you mean by continuous but not differentiable. To train the GAN, you're going to have to write a loss function that you'll have to differentiate to do gradient descent on the weights of the GAN. If that loss calculation involves non diffentiable operations, you have a problem.</p>
  <p>In practice, in neural network libraries such as JAX or PyTorch, it doesn't matter that the loss is not differentiable as long as it's piece-wise differentiable, which is the case of the <span class="math-container">$\text{argmax}$</span> function.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-31T20:08:24.157Z
Answers.LastActivityDate: 2022-08-31T20:17:03.820Z
Answers.LastEditDate: 2022-08-31T20:17:03.820Z
Answers.UserDisplayName: Raphael Lopez Kaufman
Body: <p>A GAN-like architecture for text generation is proposed in <a href="https://becominghuman.ai/generative-adversarial-networks-for-text-generation-part-1-2b886c8cab10" rel="nofollow noreferrer">'Generative Adversarial Networks for Text Generation'</a>.</p>
  <p><strong>The setup is the following:</strong></p>
  <ul>
  <li>The generator of the GAN is proposed to be a recurrent neural network that its by itself a text generator.</li>
  <li>The internal latent vectors of the GAN ( denoted by <span class="math-container">$G(\vec{z})$</span> ) are the outputs of the generator RNN.</li>
  <li>The discriminator is not made explicit; but let's imagine that is <em>&quot;any architecture that can classify text into two categories&quot;</em> (&quot;real&quot; and &quot;fake&quot; in our case).</li>
  <li>The generator RNN is trained to minimize <span class="math-container">$(1- D(G(\vec{z})))$</span> (where <span class="math-container">$D(G(\vec{z})))$</span> is the discriminator output) as usual.</li>
  </ul>
  <p>Now it comes a somewhat obscure statement:</p>
  <blockquote>
  <p>Remember that while decoding using an RNN, at every time step we make the choice of the next word by picking the word corresponding to the maximum probability from the output of the softmax function. <em><strong>This “picking” operation is non-differentiable.</strong></em></p>
  </blockquote>
  <p><strong>Question:</strong></p>
  <p><strong>What is the precise statement of that vague paragraph?</strong></p>
  <p>My expectation is to explicitly write the function that is just continuous but not differentiable.</p>
  <p><strong>Thanks in advance!</strong></p>
  
Tags: <natural-language-processing><recurrent-neural-networks><math><generative-adversarial-networks><text-generation>
Title: Clarification on GANs for text generation
UserDisplayName: Ramiro Hum-Sah
ThreadText: # Clarification on GANs for text generation
  
  (Asked by: username_0 on 2022-08-31)
  
  A GAN-like architecture for text generation is proposed in ['Generative Adversarial Networks for Text Generation'](https://becominghuman.ai/generative-adversarial-networks-for-text-generation-part-1-2b886c8cab10).
  
  **The setup is the following:**
  
  * The generator of the GAN is proposed to be a recurrent neural network that its by itself a text generator.
  * The internal latent vectors of the GAN ( denoted by $G(\vec{z})$ ) are the outputs of the generator RNN.
  * The discriminator is not made explicit; but let's imagine that is *"any architecture that can classify text into two categories"* ("real" and "fake" in our case).
  * The generator RNN is trained to minimize $(1- D(G(\vec{z})))$ (where $D(G(\vec{z})))$ is the discriminator output) as usual.
  
  Now it comes a somewhat obscure statement:
  
  > Remember that while decoding using an RNN, at every time step we make the choice of the next word by picking the word corresponding to the maximum probability from the output of the softmax function. ***This “picking” operation is non-differentiable.***
  
  **Question:**
  
  **What is the precise statement of that vague paragraph?**
  
  My expectation is to explicitly write the function that is just continuous but not differentiable.
  
  **Thanks in advance!**
  
  **Tags:** natural-language-processing, recurrent-neural-networks, math, generative-adversarial-networks, text-generation
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-08-31. Score: 3 | Accepted answer
  
  When you sample a sentence using an RNN, at each step $t$, if $N$ is the size of your vocabulary, $W$ the array with the $N$ words in your vocabulary, and $f$ your RNN without its final linear layer, you do:
  
  $$p\_{t+1} = \text{softmax}(f(w\_t))$$
  $$i\_{t+1} = \text{argmax}(p\_{t+1})$$
  $$w\_{t+1} = W[i\_{t+1}]$$
  where $p\_{t+1}$ is a vector of probabilities of size $N$, and $w\_{t+1}$ a word from the vocabulary.
  
  The $\text{argmax}$ operation to get $w\_{t+1}$ is not differentiable.
  
  I'm not sure what you mean by continuous but not differentiable. To train the GAN, you're going to have to write a loss function that you'll have to differentiate to do gradient descent on the weights of the GAN. If that loss calculation involves non diffentiable operations, you have a problem.
  
  In practice, in neural network libraries such as JAX or PyTorch, it doesn't matter that the loss is not differentiable as long as it's piece-wise differentiable, which is the case of the $\text{argmax}$ function.
  
  
  ---
  
  