Answers.Body: <p>Formally, for an input <span class="math-container">$x$</span>, <span class="math-container">$D(x)$</span> gives you the probability of <span class="math-container">$x$</span> being real. In this sense <span class="math-container">$D:\mathcal{X}\rightarrow [0,1]$</span>, where <span class="math-container">$\mathcal{X}$</span> is the input space.</p>
  <p>That said, the output of the discriminator is a probability (hence within 0 and 1), and you get the prediction (fake or real) by considering the most probable outcome. Informally <span class="math-container">$D(x) = fake$</span> for <span class="math-container">$D(x) &lt; 0.5$</span> and real otherwise.</p>
  <p><strong>@edit:</strong> on the issue of Least Squares GAN.</p>
  <p>In the first 2 paragraphs of my answer, I considered the case of the original GAN proposed by Goodfellow et al. [1]. Nonetheless other types of GANs exist, that do not employ a sigmoid activation at the output layer of the discriminator. That is the case of the Least Squares GAN of Mao et al. [2], upon which the authors of CycleGAN based themselves on [3].</p>
  <p>The authors of [2] raise the issue of using a sigmoid activation in section 3.2:</p>
  <blockquote>
  <p>when updating the generator, this loss function (cross-entropy on sigmoid activations) will cause the problem of vanishing gradients for the samples that are on the correct side of the decision boundary, but are still far from the real data.</p>
  </blockquote>
  <p>The LS-GAN proposes the following workaround: <span class="math-container">$D(x) \in \mathbb{R}$</span>, thus no sigmoid activation at the end. The loss is then substituted by the least squares loss. As follows, there is no clear encoding for fake and real labels anymore. For that reason the authors of [2] introduce constants <span class="math-container">$a$</span> and <span class="math-container">$b$</span>, such that if <span class="math-container">$D(x) \approx a \implies x$</span> is fake, and <span class="math-container">$D(x) \approx b \implies x$</span> is real. For the generator, there is another constant <span class="math-container">$c$</span> such that <span class="math-container">$D(G(z)) \approx c \implies G(z)$</span> is fake. In [3] the authors picked <span class="math-container">$a = 0$</span>, <span class="math-container">$b = 1$</span> and <span class="math-container">$c = 1$</span>.</p>
  <h2>References</h2>
  <p>[1] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., ... &amp; Bengio, Y. (2014). <a href="https://arxiv.org/pdf/1406.2661.pdf" rel="nofollow noreferrer">Generative adversarial nets</a>. Advances in neural information processing systems, 27.</p>
  <p>[2] Mao, X., Li, Q., Xie, H., Lau, R. Y., Wang, Z., &amp; Paul Smolley, S. (2017). <a href="https://openaccess.thecvf.com/content_ICCV_2017/papers/Mao_Least_Squares_Generative_ICCV_2017_paper.pdf" rel="nofollow noreferrer">Least squares generative adversarial networks</a>. In Proceedings of the IEEE international conference on computer vision (pp. 2794-2802).</p>
  <p>[3] Zhu, J. Y., Park, T., Isola, P., &amp; Efros, A. A. (2017). <a href="https://arxiv.org/pdf/1703.10593.pdf" rel="nofollow noreferrer">Unpaired image-to-image translation using cycle-consistent adversarial networks</a>. In Proceedings of the IEEE international conference on computer vision (pp. 2223-2232).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-07T07:49:04.593Z
Answers.Text: After further readings, I wanted to ask one more thing.
  Using the Sigmoid Activation function in the last Layer of the PatchGAN does push the values of the discriminator function between 0-1. Without it whole ℝ is possible as output for D(x). In the original CycleGAN Paper they removed the Sigmoid Activation function and don’t use any other activation function. So without it I don’t understand how does the adversarial loss e.g the expression (D(x) - 1) then makes any sense? If D(x) is a number in ℝ, how does reducing it by one gives me information about the real or fakeness of the image?
Answers.UserDisplayName: Lukas Pezzei
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-07T07:55:17.693Z
Answers.Text: As the authors mention in Section 4 of their paper, this is a trick for acquiring better performance. I will edit my answer to comment further on this choice.
Answers.UserDisplayName: Eduardo Montesuma
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-07T14:33:17.480Z
Answers.Text: Thank you for your detailed answer! 
  In the meantime i stumbeld upon another question regarding the adversarial loss of a CycleGAN. If I understood that correctly a LSGAN uses L2 loss or MSE loss. 
  MSE has additionally the term 1/N. Is that correctly? 
  The LSGAN Paper [2] has 1/2 in the formula in paragraph 3.2. But in the original Paper of the CycleGan [3] the same formula in Chapter 4. does not have the 1/2 upfront. Now to my Question. Where does the 1/2 come from and why is it in the CycleGAN Paper missing? 
  Again thank you very much so far for all the help!
Answers.UserDisplayName: Lukas Pezzei
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-07T15:14:01.153Z
Answers.Text: The 1/N comes from the expectation operator, as you approximate it empirically (N being the number of samples taken from a given distribution). Concerning the 1/2, it is actually entirely optional as it does not change the optimal point of a function.
Answers.UserDisplayName: Eduardo Montesuma
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-06T22:48:49.873Z
Answers.LastActivityDate: 2022-06-07T08:06:59.987Z
Answers.LastEditDate: 2022-06-07T08:06:59.987Z
Answers.UserDisplayName: Eduardo Montesuma
Answers.Body: <p>Let me try to explain this way, comment if you think it's incorrect.</p>
  <p>Assume a simple linear function, <span class="math-container">$y=f(x)=ax+b$</span> where <span class="math-container">$a \in \mathbb{R}^*$</span> and <span class="math-container">$b\in \mathbb{R}$</span>, each value of <span class="math-container">$y$</span> is unique, which means we can only get the same <span class="math-container">$y$</span> from the same <span class="math-container">$x$</span>.</p>
  <p>In GANs, the Discriminator plays the same role as <span class="math-container">$f$</span>, with the more complex function and the high dimension input (for example, an MNIST's image will have <span class="math-container">$32\times32=1024$</span> dimension), <span class="math-container">$y$</span> is no longer unique but still keeps its property, the closer <span class="math-container">$x$</span> (same domain), the closer <span class="math-container">$y$</span> value.</p>
  <p>Post-process the output of the discriminator is to adapt to the loss function, <a href="https://arxiv.org/abs/1406.2661" rel="nofollow noreferrer">the original GANs of Ian Goodfellow</a> limited the output by a sigmoid function to wrap it in a probability range to input it to a logarithm as cross-entropy. <a href="https://arxiv.org/abs/1701.07875" rel="nofollow noreferrer">WGAN</a> measures the Wasserstein distance by the real value from the discriminator or <a href="https://arxiv.org/pdf/1705.02894.pdf" rel="nofollow noreferrer">Hinge loss</a> clamps the output cap as 0.</p>
  <p>If you feel hard with the definition of the domain, take a look at the famous problem <a href="https://www.kaggle.com/c/dogs-vs-cats" rel="nofollow noreferrer">&quot;dogs vs cats&quot;</a>, it's the binary classification, and the task is to build a deep learning model to distinguish images. There are many types of dogs but they still have common characteristics that make them classified as a &quot;dog&quot; domain. In GANs, the task of the discriminator is the same, separate real and fake domains just like dog and cat.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-07T02:30:08.867Z
Answers.LastActivityDate: 2022-06-07T02:30:08.867Z
Answers.UserDisplayName: CuCaRot
Body: <p>I've read that the discriminator <span class="math-container">$D$</span> validates an image <span class="math-container">$D(x)$</span>, where <span class="math-container">$x$</span> is either a real image or a fake one created by the generator, i.e. <span class="math-container">$ D(G(x))$</span>.</p>
  <p>What does the function of the discriminator return? Is it either 0 (marked as fake) or 1 (discriminator thinks the image is real)? I have read that this function returns the whole <span class="math-container">$\mathbb{R}$</span>, but I don't understand what the output then means.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-06T22:28:32.027Z
Comments.Text: Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.
Comments.UserDisplayName: Community
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-07T02:13:19.067Z
Comments.Text: In general, the discriminator returns **probability**. But, there are some GANs in which the discriminator returns over a range of real numbers. So, both can be true and it depends on the GAN you are considering. For applications, discriminator is used as a binary classifier also;
Comments.UserDisplayName: hanugm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-07T20:21:59.567Z
Comments.Text: @hanugm That looks like an answer to me, even if it's short - some correct answers are just short. Maybe just provide some references or names of GANs that return one over the other.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-09T13:44:59.953Z
Comments.Text: Notice this is a classification problem, just like any other neural network classification problem. You could think about how a neural network tells us whether an image is a cow or a sheep.
Comments.UserDisplayName: Criticize SE actions means ban
Tags: <objective-functions><generative-adversarial-networks><loss><discriminator>
Title: What is the domain of the discriminator of a GAN?
UserDisplayName: Lukas Pezzei
ThreadText: # What is the domain of the discriminator of a GAN?
  
  (Asked by: username_0 on 2022-06-06)
  
  I've read that the discriminator $D$ validates an image $D(x)$, where $x$ is either a real image or a fake one created by the generator, i.e. $ D(G(x))$.
  
  What does the function of the discriminator return? Is it either 0 (marked as fake) or 1 (discriminator thinks the image is real)? I have read that this function returns the whole $\mathbb{R}$, but I don't understand what the output then means.
  
  **Tags:** objective-functions, generative-adversarial-networks, loss, discriminator
  
  > **Comments:**
  > **username_1**: Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.  
  > **username_2**: In general, the discriminator returns **probability**. But, there are some GANs in which the discriminator returns over a range of real numbers. So, both can be true and it depends on the GAN you are considering. For applications, discriminator is used as a binary classifier also;  
  > **username_3**: @username_2 That looks like an answer to me, even if it's short - some correct answers are just short. Maybe just provide some references or names of GANs that return one over the other.  
  > **username_4**: Notice this is a classification problem, just like any other neural network classification problem. You could think about how a neural network tells us whether an image is a cow or a sheep.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_5 on 2022-06-06. Score: 3
  
  Formally, for an input $x$, $D(x)$ gives you the probability of $x$ being real. In this sense $D:\mathcal{X}\rightarrow [0,1]$, where $\mathcal{X}$ is the input space.
  
  That said, the output of the discriminator is a probability (hence within 0 and 1), and you get the prediction (fake or real) by considering the most probable outcome. Informally $D(x) = fake$ for $D(x) < 0.5$ and real otherwise.
  
  **@edit:** on the issue of Least Squares GAN.
  
  In the first 2 paragraphs of my answer, I considered the case of the original GAN proposed by Goodfellow et al. [1]. Nonetheless other types of GANs exist, that do not employ a sigmoid activation at the output layer of the discriminator. That is the case of the Least Squares GAN of Mao et al. [2], upon which the authors of CycleGAN based themselves on [3].
  
  The authors of [2] raise the issue of using a sigmoid activation in section 3.2:
  
  > when updating the generator, this loss function (cross-entropy on sigmoid activations) will cause the problem of vanishing gradients for the samples that are on the correct side of the decision boundary, but are still far from the real data.
  
  The LS-GAN proposes the following workaround: $D(x) \in \mathbb{R}$, thus no sigmoid activation at the end. The loss is then substituted by the least squares loss. As follows, there is no clear encoding for fake and real labels anymore. For that reason the authors of [2] introduce constants $a$ and $b$, such that if $D(x) \approx a \implies x$ is fake, and $D(x) \approx b \implies x$ is real. For the generator, there is another constant $c$ such that $D(G(z)) \approx c \implies G(z)$ is fake. In [3] the authors picked $a = 0$, $b = 1$ and $c = 1$.
  
  ## References
  
  [1] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., ... & Bengio, Y. (2014). [Generative adversarial nets](https://arxiv.org/pdf/1406.2661.pdf). Advances in neural information processing systems, 27.
  
  [2] Mao, X., Li, Q., Xie, H., Lau, R. Y., Wang, Z., & Paul Smolley, S. (2017). [Least squares generative adversarial networks](https://openaccess.thecvf.com/content_ICCV_2017/papers/Mao_Least_Squares_Generative_ICCV_2017_paper.pdf). In Proceedings of the IEEE international conference on computer vision (pp. 2794-2802).
  
  [3] Zhu, J. Y., Park, T., Isola, P., & Efros, A. A. (2017). [Unpaired image-to-image translation using cycle-consistent adversarial networks](https://arxiv.org/pdf/1703.10593.pdf). In Proceedings of the IEEE international conference on computer vision (pp. 2223-2232).
  
  > **Comments:**
  > **username_0**: After further readings, I wanted to ask one more thing. Using the Sigmoid Activation function in the last Layer of the PatchGAN does push the values of the discriminator function between 0-1. Without it whole ℝ is possible as output for D(x). In the original CycleGAN Paper they removed the Sigmoid Activation function and don’t use any other activation function. So without it I don’t understand how does the adversarial loss e.g the expression (D(x) - 1) then makes any sense? If D(x) is a number in ℝ, how does reducing it by one gives me information about the real or fakeness of the image?  
  > **username_5**: As the authors mention in Section 4 of their paper, this is a trick for acquiring better performance. I will edit my answer to comment further on this choice.  
  > **username_0**: Thank you for your detailed answer!  In the meantime i stumbeld upon another question regarding the adversarial loss of a CycleGAN. If I understood that correctly a LSGAN uses L2 loss or MSE loss.  MSE has additionally the term 1/N. Is that correctly?  The LSGAN Paper [2] has 1/2 in the formula in paragraph 3.2. But in the original Paper of the CycleGan [3] the same formula in Chapter 4. does not have the 1/2 upfront. Now to my Question. Where does the 1/2 come from and why is it in the CycleGAN Paper missing?  Again thank you very much so far for all the help!  
  > **username_5**: The 1/N comes from the expectation operator, as you approximate it empirically (N being the number of samples taken from a given distribution). Concerning the 1/2, it is actually entirely optional as it does not change the optimal point of a function.  
  
  ---
  
  ### Answer by username_6 on 2022-06-07. Score: 0
  
  Let me try to explain this way, comment if you think it's incorrect.
  
  Assume a simple linear function, $y=f(x)=ax+b$ where $a \in \mathbb{R}^\*$ and $b\in \mathbb{R}$, each value of $y$ is unique, which means we can only get the same $y$ from the same $x$.
  
  In GANs, the Discriminator plays the same role as $f$, with the more complex function and the high dimension input (for example, an MNIST's image will have $32\times32=1024$ dimension), $y$ is no longer unique but still keeps its property, the closer $x$ (same domain), the closer $y$ value.
  
  Post-process the output of the discriminator is to adapt to the loss function, [the original GANs of Ian Goodfellow](https://arxiv.org/abs/1406.2661) limited the output by a sigmoid function to wrap it in a probability range to input it to a logarithm as cross-entropy. [WGAN](https://arxiv.org/abs/1701.07875) measures the Wasserstein distance by the real value from the discriminator or [Hinge loss](https://arxiv.org/pdf/1705.02894.pdf) clamps the output cap as 0.
  
  If you feel hard with the definition of the domain, take a look at the famous problem ["dogs vs cats"](https://www.kaggle.com/c/dogs-vs-cats), it's the binary classification, and the task is to build a deep learning model to distinguish images. There are many types of dogs but they still have common characteristics that make them classified as a "dog" domain. In GANs, the task of the discriminator is the same, separate real and fake domains just like dog and cat.
  
  
  ---
  
  