Answers.Body: <p>Figure 3 in the <a href="http://proceedings.mlr.press/v70/arjovsky17a/arjovsky17a.pdf" rel="nofollow noreferrer">original WGAN paper</a> is actually quite helpful to understand the difference between the score in WGAN and the probability in GAN (see screenshot below). The blue distribution are real samples, and the green one are fake samples. The Vanilla GAN trained in this example identifies the real samples as '100% real' (red curve) and the fake samples as '100% fake'. This leads to the problem of vanishing gradients and the well-known mode collapse of original GANs.</p>
  <p>The Wasserstein GAN, on the other hand, gives each sample a score. The benefit of the score is that we can now identify samples that are more likely real than others, or more likely fake. For example, the further to the left a distribution is located, the more negative the WGAN score will be. We have therefore a continuum that doesn't end in 0 and 1, but can compare between samples that are 'good' and those which are 'better'. A normal GAN would identify both as 'good', making further improvement difficult.</p>
  <p><a href="https://i.sstatic.net/0YHLW.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/0YHLW.png" alt="Figure 3 from Arjovsky et al, 2017" /></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-27T10:06:10.090Z
Answers.Text: Thanks! There is an aspect which is not clear to me, if a binary cross entropy is used as activation for the last layer of the discriminator you will obtain for each sample a probability for belonging to a class. So "the further to the left a distribution is located" the lower will be the probability for the samples to belong to the fake class. The only problem that could come to mind is that the classifier is so sure of the choice, since in this case the samples are even linearly separable, which will always return probability about 1 for one of the two classes, is it correct?
Answers.UserDisplayName: Stefano Barone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-28T00:23:20.153Z
Answers.Text: Yes! That's at least one of the reasons that lead to mode collapse in traditional GANs, i.e. the inability to recover a multi-modal distribution (see example in Figure 2 of the WGAN paper). If the generator outputs one of the modes, but the samples are already judged as "very real looking" (that is, "very far on the left"), there is no motivation for it to improve as the gradients are vanishing and don't give useful information. If instead the Wasserstein distance is used, which can be identified as a score, we don't have this problem but a monotonous curve with useful gradients.
Answers.UserDisplayName: Mafu
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-26T10:08:58.840Z
Answers.LastActivityDate: 2021-01-26T10:08:58.840Z
Answers.UserDisplayName: Mafu
Body: <p>I am quite new to GAN and I am reading about WGAN vs DCGAN.</p>
  <p>Relating to the Wasserstein GAN (WGAN), I read <a href="https://machinelearningmastery.com/how-to-code-a-wasserstein-generative-adversarial-network-wgan-from-scratch/" rel="nofollow noreferrer">here</a></p>
  <blockquote>
  <p>Instead of using a discriminator to classify or predict the <strong>probability</strong> of generated images as being real or fake, the WGAN changes or replaces the discriminator model with a critic that <strong>scores</strong> the realness or fakeness of a given image.</p>
  </blockquote>
  <p>In practice, I don't understand what the difference is between a <em>score</em> of the realness or fakeness of a given image and a <em>probability</em> that the generated images are real or fake.</p>
  <p>Aren't scores probabilities?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-25T13:10:28.090Z
Comments.Text: I didn't read that paper and I don't have time now to read it, but the term _score_ in statistics (and machine learning) sometimes refers to a specific concept. Take a look at [this](https://en.wikipedia.org/wiki/Score_(statistics)). If nobody provides an answer, you may want to read [the WGAN paper](http://proceedings.mlr.press/v70/arjovsky17a.html) (to know how they defined the _score_ there) and the linked Wikipedia article, and then provide a formal answer to your own question below ;)
Comments.UserDisplayName: nbro
Tags: <deep-learning><terminology><generative-adversarial-networks><probability><wasserstein-gan>
Title: Aren't scores in the Wasserstein GAN probabilities?
UserDisplayName: Stefano Barone
ThreadText: # Aren't scores in the Wasserstein GAN probabilities?
  
  (Asked by: username_0 on 2021-01-25)
  
  I am quite new to GAN and I am reading about WGAN vs DCGAN.
  
  Relating to the Wasserstein GAN (WGAN), I read [here](https://machinelearningmastery.com/how-to-code-a-wasserstein-generative-adversarial-network-wgan-from-scratch/)
  
  > Instead of using a discriminator to classify or predict the **probability** of generated images as being real or fake, the WGAN changes or replaces the discriminator model with a critic that **scores** the realness or fakeness of a given image.
  
  In practice, I don't understand what the difference is between a *score* of the realness or fakeness of a given image and a *probability* that the generated images are real or fake.
  
  Aren't scores probabilities?
  
  **Tags:** deep-learning, terminology, generative-adversarial-networks, probability, wasserstein-gan
  > **Comments:**
  > **username_1**: I didn't read that paper and I don't have time now to read it, but the term _score_ in statistics (and machine learning) sometimes refers to a specific concept. Take a look at [this](https://en.wikipedia.org/wiki/Score_(statistics)). If nobody provides an answer, you may want to read [the WGAN paper](http://proceedings.mlr.press/v70/arjovsky17a.html) (to know how they defined the _score_ there) and the linked Wikipedia article, and then provide a formal answer to your own question below ;)  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-01-26. Score: 1 | Accepted answer
  
  Figure 3 in the [original WGAN paper](http://proceedings.mlr.press/v70/arjovsky17a/arjovsky17a.pdf) is actually quite helpful to understand the difference between the score in WGAN and the probability in GAN (see screenshot below). The blue distribution are real samples, and the green one are fake samples. The Vanilla GAN trained in this example identifies the real samples as '100% real' (red curve) and the fake samples as '100% fake'. This leads to the problem of vanishing gradients and the well-known mode collapse of original GANs.
  
  The Wasserstein GAN, on the other hand, gives each sample a score. The benefit of the score is that we can now identify samples that are more likely real than others, or more likely fake. For example, the further to the left a distribution is located, the more negative the WGAN score will be. We have therefore a continuum that doesn't end in 0 and 1, but can compare between samples that are 'good' and those which are 'better'. A normal GAN would identify both as 'good', making further improvement difficult.
  
  [![Figure 3 from Arjovsky et al, 2017](https://i.sstatic.net/0YHLW.png)](https://i.sstatic.net/0YHLW.png)
  
  > **Comments:**
  > **username_0**: Thanks! There is an aspect which is not clear to me, if a binary cross entropy is used as activation for the last layer of the discriminator you will obtain for each sample a probability for belonging to a class. So "the further to the left a distribution is located" the lower will be the probability for the samples to belong to the fake class. The only problem that could come to mind is that the classifier is so sure of the choice, since in this case the samples are even linearly separable, which will always return probability about 1 for one of the two classes, is it correct?  
  > **username_2**: Yes! That's at least one of the reasons that lead to mode collapse in traditional GANs, i.e. the inability to recover a multi-modal distribution (see example in Figure 2 of the WGAN paper). If the generator outputs one of the modes, but the samples are already judged as "very real looking" (that is, "very far on the left"), there is no motivation for it to improve as the gradients are vanishing and don't give useful information. If instead the Wasserstein distance is used, which can be identified as a score, we don't have this problem but a monotonous curve with useful gradients.  
  
  ---
  
  