Answers.Body: <p>I know that this is an old question, but here are my two cents.</p>
  <ol>
  <li>Yes, it is possible that the Wasserstein Loss goes negative.</li>
  <li>Yes the implementation seems to be correct. However, it also depends on the implementation of the generator loss. You can implement it in an inverted manner, and it would also work. The key is to ensure that the generator and the discriminator compete <em>against</em> one another, not with one another.</li>
  <li>Interpretation of the Wasserstein loss is a complex undertaking, and you should rarely make conclusions just based on whether the loss is going up or down. If have seen WGANs converge with all kinds of negative and positive losses in the critic. There are some resources out there that give you some tips on how to interpret the loss. However, the convergence of the loss highly depends on a bunch of factors such as the dataset, the type and size of the architecture, the learning rate and many more. You should assume that what works in the video on one dataset does not translate to any meaningful insights for training on another dataset with a different architecture.</li>
  <li>If you have exactly the same implementation and dataset, you <em>should</em> see approximately the same behaviour. However, note that through updates on libraries/differences across libraries, those changes might be obscure. These can involve the initialization of weights or default parameters of often-used functionalities.</li>
  <li>You should abstain from trying to compare different runs based on the trajectory of the loss. Please use a separate metric for that.</li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-12T08:42:58.963Z
Answers.LastActivityDate: 2023-10-12T08:42:58.963Z
Answers.UserDisplayName: Robin van Hoorn
Body: <p>The <a href="https://arxiv.org/pdf/1701.07875.pdf" rel="noreferrer">WGAN paper</a> concretely proposes Algorithm 1 (cf. page 8). Now, they also state what their loss for the critic and the generator is.</p>
  <p>When implementing the critic loss (so lines 5 and 6 of Algorithm 1), they maximize the parameters <span class="math-container">$w$</span> (instead of minimizing them as one would normally do) by writing <span class="math-container">$w \leftarrow w + \alpha \cdot \text{RMSProp}\left(w, g_w \right)$</span>. Their loss seems to be <span class="math-container">$$\frac{1}{m}\sum_{i = 1}^{m}f_{w}\left(x^{\left(i\right)} \right) - \frac{1}{m}\sum_{i = 1}^{m} f_{w}\left( g_{\theta}\left( z^{\left( i\right)}\right)\right).\quad \quad (1)$$</span></p>
  <p>The function <span class="math-container">$f$</span> is the critic, i.e. a neural network, and the way this loss is implemented in PyTorch in <a href="https://www.youtube.com/watch?v=pG0QZ7OddX4" rel="noreferrer">this youtbe video</a> (cf. minutes 11:00 to 12:26) is as follows:</p>
  <p><code>critic_real = critic(real_images)</code></p>
  <p><code>critic_fake = critic(generator(noise))</code></p>
  <p><code>loss_critic = -(torch.mean(critic_real) - torch.mean(critic_fake))</code></p>
  <p><strong>My question is:</strong> In my own experiments with the <a href="http://mmlab.ie.cuhk.edu.hk/projects/CelebA.html" rel="noreferrer">CelebA dataset</a>, I found that the critic loss is negative, and that the quality of the images is better if the negative critic loss is <em>higher</em> instead of lower, so <span class="math-container">$-0.75$</span> for the critic loss resulted in better generated iamges than a critic loss of <span class="math-container">$-1.26$</span> e.g.</p>
  <p>Is there an error in the implementation in the youtube video of Eq. (1) and Algorithm 1 of the WGAN paper maybe? In my opinion, the implementation in the video is correct, but I am still confused then on why I get better images when the loss is higher ...</p>
  <p>Cheers!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-28T15:46:59.250Z
Comments.Text: Did you observe better images also with higher positive values? The Wasserstein loss is supposed to converge to 0, since it is basically the Wasserstein distance between the distribution of real images and fake generated ones. So I would say that in the example you provide it is normal to get better images with a critic loss of -.75, closer to 0 than -1.26, which tells you the discriminator is having a hard time distinguishing between fake and real images (hence better quality).
Comments.UserDisplayName: Edoardo Guerriero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-29T20:53:40.167Z
Comments.Text: Hi Edoardo, okay, I see your point. Given that $-0.75$ is closer to $0$ than $-1.26$, it is good to see that I see better images, I guess. The only thing interesting/slightly confusing me is that I haven't a positive loss for the critic yet, is that normal?
Comments.UserDisplayName: Anonymous5638
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-29T23:14:46.597Z
Comments.Text: In my experience it is possible to get negative scores using the Wasserstein loss. Again, cause rather than a usual loss the scores represent a distance between two means, that the discriminator tries to maximize. Negative scores simply means that the mean of the distribution of the generated images is bigger than the mean of the distribution of the real images. Positive scores means the other way around, but either way the game of the discriminator is just to maximize the distance, i.e. increase the absolute value, regardless of the sign in front of it.
Comments.UserDisplayName: Edoardo Guerriero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-30T12:35:44.203Z
Comments.Text: Just a last advice: looking at the generator and discriminator losses is obviously important, but in my experience only for understanding if the training is stable or not. But ultimately, what you care about is the quality of the images produced, and you should implement some specific metrics to evaluate that, and check mostly those metrics to perform hyperparameters tuning. Don't waste too much time trying to get 'good values' on the discriminator and generator losses.
Comments.UserDisplayName: Edoardo Guerriero
Tags: <papers><generative-adversarial-networks><implementation><wasserstein-gan>
Title: Wasserstein GAN: Implemention of Critic Loss Correct?
UserDisplayName: Anonymous5638
ThreadText: # Wasserstein GAN: Implemention of Critic Loss Correct?
  
  (Asked by: username_0 on 2020-12-27)
  
  The [WGAN paper](https://arxiv.org/pdf/1701.07875.pdf) concretely proposes Algorithm 1 (cf. page 8). Now, they also state what their loss for the critic and the generator is.
  
  When implementing the critic loss (so lines 5 and 6 of Algorithm 1), they maximize the parameters $w$ (instead of minimizing them as one would normally do) by writing $w \leftarrow w + \alpha \cdot \text{RMSProp}\left(w, g\_w \right)$. Their loss seems to be $$\frac{1}{m}\sum\_{i = 1}^{m}f\_{w}\left(x^{\left(i\right)} \right) - \frac{1}{m}\sum\_{i = 1}^{m} f\_{w}\left( g\_{\theta}\left( z^{\left( i\right)}\right)\right).\quad \quad (1)$$
  
  The function $f$ is the critic, i.e. a neural network, and the way this loss is implemented in PyTorch in [this youtbe video](https://www.youtube.com/watch?v=pG0QZ7OddX4) (cf. minutes 11:00 to 12:26) is as follows:
  
  `critic_real = critic(real_images)`
  
  `critic_fake = critic(generator(noise))`
  
  `loss_critic = -(torch.mean(critic_real) - torch.mean(critic_fake))`
  
  **My question is:** In my own experiments with the [CelebA dataset](http://mmlab.ie.cuhk.edu.hk/projects/CelebA.html), I found that the critic loss is negative, and that the quality of the images is better if the negative critic loss is *higher* instead of lower, so $-0.75$ for the critic loss resulted in better generated iamges than a critic loss of $-1.26$ e.g.
  
  Is there an error in the implementation in the youtube video of Eq. (1) and Algorithm 1 of the WGAN paper maybe? In my opinion, the implementation in the video is correct, but I am still confused then on why I get better images when the loss is higher ...
  
  Cheers!
  
  **Tags:** papers, generative-adversarial-networks, implementation, wasserstein-gan
  > **Comments:**
  > **username_1**: Did you observe better images also with higher positive values? The Wasserstein loss is supposed to converge to 0, since it is basically the Wasserstein distance between the distribution of real images and fake generated ones. So I would say that in the example you provide it is normal to get better images with a critic loss of -.75, closer to 0 than -1.26, which tells you the discriminator is having a hard time distinguishing between fake and real images (hence better quality).  
  > **username_0**: Hi Edoardo, okay, I see your point. Given that $-0.75$ is closer to $0$ than $-1.26$, it is good to see that I see better images, I guess. The only thing interesting/slightly confusing me is that I haven't a positive loss for the critic yet, is that normal?  
  > **username_1**: In my experience it is possible to get negative scores using the Wasserstein loss. Again, cause rather than a usual loss the scores represent a distance between two means, that the discriminator tries to maximize. Negative scores simply means that the mean of the distribution of the generated images is bigger than the mean of the distribution of the real images. Positive scores means the other way around, but either way the game of the discriminator is just to maximize the distance, i.e. increase the absolute value, regardless of the sign in front of it.  
  > **username_1**: Just a last advice: looking at the generator and discriminator losses is obviously important, but in my experience only for understanding if the training is stable or not. But ultimately, what you care about is the quality of the images produced, and you should implement some specific metrics to evaluate that, and check mostly those metrics to perform hyperparameters tuning. Don't waste too much time trying to get 'good values' on the discriminator and generator losses.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-10-12. Score: 0
  
  I know that this is an old question, but here are my two cents.
  
  1. Yes, it is possible that the Wasserstein Loss goes negative.
  2. Yes the implementation seems to be correct. However, it also depends on the implementation of the generator loss. You can implement it in an inverted manner, and it would also work. The key is to ensure that the generator and the discriminator compete *against* one another, not with one another.
  3. Interpretation of the Wasserstein loss is a complex undertaking, and you should rarely make conclusions just based on whether the loss is going up or down. If have seen WGANs converge with all kinds of negative and positive losses in the critic. There are some resources out there that give you some tips on how to interpret the loss. However, the convergence of the loss highly depends on a bunch of factors such as the dataset, the type and size of the architecture, the learning rate and many more. You should assume that what works in the video on one dataset does not translate to any meaningful insights for training on another dataset with a different architecture.
  4. If you have exactly the same implementation and dataset, you *should* see approximately the same behaviour. However, note that through updates on libraries/differences across libraries, those changes might be obscure. These can involve the initialization of weights or default parameters of often-used functionalities.
  5. You should abstain from trying to compare different runs based on the trajectory of the loss. Please use a separate metric for that.
  
  
  ---
  
  