Answers.Body: <p>Your statement that  we should be expecting some similarity in graphical patterns is not correct. The GAN loss takes the following form:</p>
  <p><span class="math-container">$$
  \min_G \max_D V(D,G) = \mathbb{E}_{x \sim p_{data}(x)} [\log D(x)] + \\
  + \mathbb{E}_{z \sim p_z(z)} [\log (1 - D(G(z)))]\label{minimax}
  $$</span></p>
  <p>We want to maximize this loss w.r.t. D in order to distinguish between real and fake samples (<span class="math-container">$D(x)\rightarrow 1$</span> and <span class="math-container">$D(G(z))\rightarrow 0$</span>), whereas the task of G is exactly the <strong>opposite</strong>. We want to minimize the function w.r.t. G so that the difference between real and generated data will be minimal. Thus, the problem becomes a minimax non-cooperative game. Here is a <a href="https://danieltakeshi.github.io/2017/03/05/understanding-generative-adversarial-networks/" rel="nofollow noreferrer">good explanation</a> of the loss.</p>
  <p>Training GANs requires finding a Nash equilibrium of a non-convex game with continuous, high dimensional parameters. GANs are typically trained using gradient descent techniques that are designed to find a low value of a cost function, rather than to find the Nash equilibrium of a game. Two models are trained simultaneously and each model updates its cost independently with no respect to another player in the game.</p>
  <p>In other words, as soon as D becomes better, G becomes also better. Updating the gradient of both models concurrently cannot guarantee a convergence. For more information, see <a href="https://arxiv.org/abs/1606.03498" rel="nofollow noreferrer">Improved Techniques for Training GANs</a> and <a href="https://arxiv.org/abs/1701.07875" rel="nofollow noreferrer">Wasserstein GAN</a>.</p>
  <p>Moreover, since D and G play a non-cooperative game, it can be shown that there are some cases when it is impossible to find a Nash equilibrium at all:</p>
  <p><a href="https://i.sstatic.net/Jm8Y7.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Jm8Y7.png" alt="enter image description here" /></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-13T09:18:32.447Z
Answers.LastActivityDate: 2020-11-13T10:27:03.070Z
Answers.LastEditDate: 2020-11-13T10:27:03.070Z
Answers.UserDisplayName: Aray Karjauv
Body: <p>In GAN (generative adversarial networks), let us take &quot;binary cross-entropy&quot; as the loss function for discriminator <span class="math-container">$$(overall \; loss = -\sum log(D(x_i)) -\sum log(1-D(G(z_i))) $$</span>
  <span class="math-container">$$ where \; x_i = real \; image \; pixel \; matrix$$</span>
  <span class="math-container">$$ and \; z_i = a \; vector \; from \; latent \; space$$</span>.
  Let us define discriminator real loss and fake loss:
  <span class="math-container">$$ d_{fake \; loss} = -\sum log(1-D(G(Z)))$$</span>
  <span class="math-container">$$ d_{real \; loss} = -\sum log(D(x))$$</span>
  <span class="math-container">$$ d_{fake \; loss} \; implies \; discriminator \; loss \; against \; fake \; images$$</span>
  <span class="math-container">$$ d_{real \; loss} \; implies \; discriminator \; loss \; against \; real \; images$$</span>
  Generator Loss :
  <span class="math-container">$$ g_{loss} = -\sum log(D(G(z_i)))$$</span>
  Since the functions are similar, we should be expecting some similarity in graphical patterns (i.e since none of the functions are inherently oscillatory, I expect that if one comes out to be oscillatory, the other one should be the same as well). But, If you refer to chapter 10 of the book &quot;Generative Adversarial Networks with python by Jason Brownlee&quot;, we find some difference. The following are the graphs published in the book
  <a href="https://i.sstatic.net/KG6O1.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/KG6O1.png" alt="Line plots for loss and accuracy for a stable GAN (1st GRAPH plots the losses)" /></a></p>
  <p><a href="https://i.sstatic.net/5yPOF.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/5yPOF.png" alt="Line plots for loss and accuracy for a stable GAN with mode collapse" /></a></p>
  <p>Can anyone explain the difference in the plots between discriminator fake loss and generator loss (mathematically)?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-12T10:41:59.253Z
Comments.Text: You can use latex on this site. So, please, edit your post to format it with latex.
Comments.UserDisplayName: nbro
Tags: <objective-functions><generative-adversarial-networks>
Title: Explain the difference in graphical patterns between discriminator fake loss and generator loss in GAN
UserDisplayName: Howard Wolowitz
ThreadText: # Explain the difference in graphical patterns between discriminator fake loss and generator loss in GAN
  
  (Asked by: username_0 on 2020-11-11)
  
  In GAN (generative adversarial networks), let us take "binary cross-entropy" as the loss function for discriminator $$(overall \; loss = -\sum log(D(x\_i)) -\sum log(1-D(G(z\_i))) $$
  $$ where \; x\_i = real \; image \; pixel \; matrix$$
  $$ and \; z\_i = a \; vector \; from \; latent \; space$$.
  Let us define discriminator real loss and fake loss:
  $$ d\_{fake \; loss} = -\sum log(1-D(G(Z)))$$
  $$ d\_{real \; loss} = -\sum log(D(x))$$
  $$ d\_{fake \; loss} \; implies \; discriminator \; loss \; against \; fake \; images$$
  $$ d\_{real \; loss} \; implies \; discriminator \; loss \; against \; real \; images$$
  Generator Loss :
  $$ g\_{loss} = -\sum log(D(G(z\_i)))$$
  Since the functions are similar, we should be expecting some similarity in graphical patterns (i.e since none of the functions are inherently oscillatory, I expect that if one comes out to be oscillatory, the other one should be the same as well). But, If you refer to chapter 10 of the book "Generative Adversarial Networks with python by Jason Brownlee", we find some difference. The following are the graphs published in the book
  [![Line plots for loss and accuracy for a stable GAN (1st GRAPH plots the losses)](https://i.sstatic.net/KG6O1.png)](https://i.sstatic.net/KG6O1.png)
  
  [![Line plots for loss and accuracy for a stable GAN with mode collapse](https://i.sstatic.net/5yPOF.png)](https://i.sstatic.net/5yPOF.png)
  
  Can anyone explain the difference in the plots between discriminator fake loss and generator loss (mathematically)?
  
  **Tags:** objective-functions, generative-adversarial-networks
  > **Comments:**
  > **username_1**: You can use latex on this site. So, please, edit your post to format it with latex.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-11-13. Score: 1
  
  Your statement that we should be expecting some similarity in graphical patterns is not correct. The GAN loss takes the following form:
  
  $$
  \min\_G \max\_D V(D,G) = \mathbb{E}\_{x \sim p\_{data}(x)} [\log D(x)] + \\
  + \mathbb{E}\_{z \sim p\_z(z)} [\log (1 - D(G(z)))]\label{minimax}
  $$
  
  We want to maximize this loss w.r.t. D in order to distinguish between real and fake samples ($D(x)\rightarrow 1$ and $D(G(z))\rightarrow 0$), whereas the task of G is exactly the **opposite**. We want to minimize the function w.r.t. G so that the difference between real and generated data will be minimal. Thus, the problem becomes a minimax non-cooperative game. Here is a [good explanation](https://danieltakeshi.github.io/2017/03/05/understanding-generative-adversarial-networks/) of the loss.
  
  Training GANs requires finding a Nash equilibrium of a non-convex game with continuous, high dimensional parameters. GANs are typically trained using gradient descent techniques that are designed to find a low value of a cost function, rather than to find the Nash equilibrium of a game. Two models are trained simultaneously and each model updates its cost independently with no respect to another player in the game.
  
  In other words, as soon as D becomes better, G becomes also better. Updating the gradient of both models concurrently cannot guarantee a convergence. For more information, see [Improved Techniques for Training GANs](https://arxiv.org/abs/1606.03498) and [Wasserstein GAN](https://arxiv.org/abs/1701.07875).
  
  Moreover, since D and G play a non-cooperative game, it can be shown that there are some cases when it is impossible to find a Nash equilibrium at all:
  
  [![enter image description here](https://i.sstatic.net/Jm8Y7.png)](https://i.sstatic.net/Jm8Y7.png)
  
  
  ---
  
  