Answers.Body: <blockquote>
  <p>What am I missing?</p>
  </blockquote>
  <p>The generator's <em>loss</em> is not calculated by comparing its output to a target value, but by processing it through the discriminator. So it is still technically a function of its output, but a complex and indirect one.</p>
  <p>There <strong>will</strong> consequently be a gradient at the generator's output, and you could potentially derive some kind of per-pixel loss metric using that, and perhaps summarise it into a single mean value. However, I don't think that would give you any useful extra information for monitoring progress. The objective function used for training is still &quot;how well can the discriminator tell that the generator's image is not real?&quot;.</p>
  <p>Either way, <a href="https://machinelearningmastery.com/how-to-evaluate-generative-adversarial-networks/" rel="nofollow noreferrer">progress when training GANs is hard to assess</a>. You don't want to see very low loss values for discriminator or generator because they imply instability. You may be able to use a scoring system like <em>Inception Score</em>, or <em>Fréchet Inception Distance</em> which use a trained image classifier to assess quality of the output independently of generator or discriminator scores.</p>
  <p>GANs are not the only neural network architecture and training scheme where the objective function is a complex processing of the output layer. Face recognition using triplet loss also has an indirect function (this time not via a separate network). In both this case, and with GANs, you rely on being able to back-propagate through the objective function to get gradients that affect the network you are training.</p>
  <p>The take away is that objective functions do not have to be limited to direct comparisons of a neural network's output with some target value. They can be more complex than that, and this can allow invention of new types of supervised and semi-supervised training feedback.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-24T21:28:02.057Z
Answers.Text: Thanks for your response! It seems you are you saying that the gradient is derived from the discrimnator's loss. So I can almost think of the generator and the discriminator as one big combined network. Is this correct?
Answers.UserDisplayName: zacoons
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-25T07:18:49.053Z
Answers.Text: @Zacoons sort of, yes. The generator gradient is not calculated from discriminator's loss, but it does require processing by the discriminator. It's kind of the opposite of the discriminator loss, although it doesn't evaluate any real images, only the fake ones. Absolutely correct on "one big network", if you add that the discriminator's weights are not updated for generator loss
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-25T22:59:09.673Z
Answers.Text: I think I understand now. I've managed to make a simple implementation and it works somewhat (though that doesn't necessarily mean I'm doing it right). Your help is much appreciated (I'd upvote but I need >15rep). God bless.
Answers.UserDisplayName: zacoons
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-24T13:09:05.873Z
Answers.LastActivityDate: 2023-09-24T13:17:37.723Z
Answers.LastEditDate: 2023-09-24T13:17:37.723Z
Answers.UserDisplayName: Neil Slater
Body: <p>I've been reading about GANs so I can implement a simple image generator. It seems that the loss for the generator is given by the following equation: <code>log(1 - D(G(z)))</code></p>
  <p>But I don't see how this equation can apply to the generator. The generator's output is in latent space (e.g. 784) but the discriminator's output is in label space (e.g. 10). Therefore this loss would have 10 dimensions. So how can the generator, which outputs 784 dimensions, be hooked up to minimise this 10 dimensional function? What am I missing?</p>
  
Tags: <machine-learning><generative-adversarial-networks>
Title: How the generator loss works in a GAN
UserDisplayName: zacoons
ThreadText: # How the generator loss works in a GAN
  
  (Asked by: username_0 on 2023-09-24)
  
  I've been reading about GANs so I can implement a simple image generator. It seems that the loss for the generator is given by the following equation: `log(1 - D(G(z)))`
  
  But I don't see how this equation can apply to the generator. The generator's output is in latent space (e.g. 784) but the discriminator's output is in label space (e.g. 10). Therefore this loss would have 10 dimensions. So how can the generator, which outputs 784 dimensions, be hooked up to minimise this 10 dimensional function? What am I missing?
  
  **Tags:** machine-learning, generative-adversarial-networks
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-09-24. Score: 1 | Accepted answer
  
  > What am I missing?
  
  The generator's *loss* is not calculated by comparing its output to a target value, but by processing it through the discriminator. So it is still technically a function of its output, but a complex and indirect one.
  
  There **will** consequently be a gradient at the generator's output, and you could potentially derive some kind of per-pixel loss metric using that, and perhaps summarise it into a single mean value. However, I don't think that would give you any useful extra information for monitoring progress. The objective function used for training is still "how well can the discriminator tell that the generator's image is not real?".
  
  Either way, [progress when training GANs is hard to assess](https://machinelearningmastery.com/how-to-evaluate-generative-adversarial-networks/). You don't want to see very low loss values for discriminator or generator because they imply instability. You may be able to use a scoring system like *Inception Score*, or *Fréchet Inception Distance* which use a trained image classifier to assess quality of the output independently of generator or discriminator scores.
  
  GANs are not the only neural network architecture and training scheme where the objective function is a complex processing of the output layer. Face recognition using triplet loss also has an indirect function (this time not via a separate network). In both this case, and with GANs, you rely on being able to back-propagate through the objective function to get gradients that affect the network you are training.
  
  The take away is that objective functions do not have to be limited to direct comparisons of a neural network's output with some target value. They can be more complex than that, and this can allow invention of new types of supervised and semi-supervised training feedback.
  
  > **Comments:**
  > **username_0**: Thanks for your response! It seems you are you saying that the gradient is derived from the discrimnator's loss. So I can almost think of the generator and the discriminator as one big combined network. Is this correct?  
  > **username_1**: @Zacoons sort of, yes. The generator gradient is not calculated from discriminator's loss, but it does require processing by the discriminator. It's kind of the opposite of the discriminator loss, although it doesn't evaluate any real images, only the fake ones. Absolutely correct on "one big network", if you add that the discriminator's weights are not updated for generator loss  
  > **username_0**: I think I understand now. I've managed to make a simple implementation and it works somewhat (though that doesn't necessarily mean I'm doing it right). Your help is much appreciated (I'd upvote but I need >15rep). God bless.  
  
  ---
  
  