Answers.Body: <p>A function can be optimized, even if it has two inputs!</p>
  <p>First, note that the problem in question already occurs with traditional GANs as well. It might be easier to understand what is going on if you have an 'implementation' example. You can check <a href="https://github.com/rdvhoorn/GAN-Pytorch/blob/main/A-Z%20GAN%20tutorial.ipynb" rel="nofollow noreferrer">my tutorial on GANs on GitHub</a>.</p>
  <p>Specifically, for training the discriminator we indeed take to inputs:</p>
  <ol>
  <li>A batch of synthetic samples <span class="math-container">$\hat x$</span>. This we can create by taking a batch of noise vectors, which we pass through the generator (<span class="math-container">$\hat x = G(z)$</span>).</li>
  <li>A batch of real training samples <span class="math-container">$x$</span> from the dataset.</li>
  </ol>
  <p>Now we pass both through the discriminator. Obtaining both <span class="math-container">$D(\hat x)$</span> and <span class="math-container">$D(x)$</span>.</p>
  <p>Now we can calculate the loss for both outputs and combining them. For WGAN, this means taking the mean over the outputs, and subtracting them, so <span class="math-container">$mean(D(x)) - mean(D(\hat x))$</span>. This we can backpropagate, as it is simply one number. Note that backpropagating one of these terms (i.e. just <span class="math-container">$mean(D(x))$</span> ) is very similar to backpropagating both of them, as it is only one addition operation different. You need just the same amount of calculus for the one as for the other.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-08T21:32:45.100Z
Answers.Text: Thank you for your efforts. I looked through your repo and found out that I can add two cost functions together for backwards propagation. After some more research I learned that this is equivalent to backwards propagating twice. e.g. ```(loss1 + loss2).backprop(); optimizer.step();``` is the same as ```loss1.backprop(); loss2.backprop(); optimizer.step();``` God bless.
Answers.UserDisplayName: zacoons
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-09T07:35:45.820Z
Answers.Text: They are approximately the same, but your second option will train 'faster' because the learning rate is 'doubled' (applied two times). Operationally, it is the same.
Answers.UserDisplayName: Robin van Hoorn
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-07T12:33:17.703Z
Answers.LastActivityDate: 2023-10-07T12:33:17.703Z
Answers.UserDisplayName: Robin van Hoorn
Body: <p>The generator tries to maximise this function <code>D(G(z))</code>. That much I understand.</p>
  <p>But how can the critic maximise <code>D(x) - D(G(z))</code>. This has two inputs: a real instance and a noise sample. The model takes an input in order to backprop, otherwise it wouldn't be able to update the first layer. So which input do I choose? Am I missing something or does this require multi-variable calculus?</p>
  
Tags: <machine-learning><generative-adversarial-networks><wasserstein-gan>
Title: Do WGAN gradients require multi-variable calculus?
UserDisplayName: zacoons
ThreadText: # Do WGAN gradients require multi-variable calculus?
  
  (Asked by: username_0 on 2023-10-06)
  
  The generator tries to maximise this function `D(G(z))`. That much I understand.
  
  But how can the critic maximise `D(x) - D(G(z))`. This has two inputs: a real instance and a noise sample. The model takes an input in order to backprop, otherwise it wouldn't be able to update the first layer. So which input do I choose? Am I missing something or does this require multi-variable calculus?
  
  **Tags:** machine-learning, generative-adversarial-networks, wasserstein-gan
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-10-07. Score: 1
  
  A function can be optimized, even if it has two inputs!
  
  First, note that the problem in question already occurs with traditional GANs as well. It might be easier to understand what is going on if you have an 'implementation' example. You can check [my tutorial on GANs on GitHub](https://github.com/rdvhoorn/GAN-Pytorch/blob/main/A-Z%20GAN%20tutorial.ipynb).
  
  Specifically, for training the discriminator we indeed take to inputs:
  
  1. A batch of synthetic samples $\hat x$. This we can create by taking a batch of noise vectors, which we pass through the generator ($\hat x = G(z)$).
  2. A batch of real training samples $x$ from the dataset.
  
  Now we pass both through the discriminator. Obtaining both $D(\hat x)$ and $D(x)$.
  
  Now we can calculate the loss for both outputs and combining them. For WGAN, this means taking the mean over the outputs, and subtracting them, so $mean(D(x)) - mean(D(\hat x))$. This we can backpropagate, as it is simply one number. Note that backpropagating one of these terms (i.e. just $mean(D(x))$ ) is very similar to backpropagating both of them, as it is only one addition operation different. You need just the same amount of calculus for the one as for the other.
  
  > **Comments:**
  > **username_0**: Thank you for your efforts. I looked through your repo and found out that I can add two cost functions together for backwards propagation. After some more research I learned that this is equivalent to backwards propagating twice. e.g. ```(loss1 + loss2).backprop(); optimizer.step();``` is the same as ```loss1.backprop(); loss2.backprop(); optimizer.step();``` God bless.  
  > **username_1**: They are approximately the same, but your second option will train 'faster' because the learning rate is 'doubled' (applied two times). Operationally, it is the same.  
  
  ---
  
  