Answers.Body: <p>I doubt whether your derivation is correct. Your are trying to apply binary-cross entropy for data on each label separately, which is not the correct way to do.</p>
  <p>The procedure for calculating binary cross entropy is as follows</p>
  <ol>
  <li>Pass the input <span class="math-container">$x$</span> whose label is <span class="math-container">$y \in \{0, 1\}$</span> to your model <span class="math-container">$M$</span>.</li>
  <li>Obtain <span class="math-container">$\hat{y} \in [0, 1]$</span> as output of your model <span class="math-container">$M$</span> instead of actual label <span class="math-container">$y$</span>.</li>
  <li>Calculate binary cross-entropy loss using the equation</li>
  </ol>
  <p><span class="math-container">$$L_{CE} = y \log \hat{y} + (1-y) \log (1 - \hat{y})$$</span></p>
  <p>It is true that there are two types of inputs to a discriminator: genuine and fake. Genuine data is labelled by 1 and fake data is labelled by 0. Use the variable <span class="math-container">$x'$</span> to represent the input to the discriminator module <span class="math-container">$D$</span>. If the input <span class="math-container">$x'$</span> is genuine then its label is 1 and if your input <span class="math-container">$x'$</span> is fake then its label is 0. Note that it is better to avoid the unnecessary details regarding the generator or noise vector while formulating the binary cross-entropy loss of discriminator. Just see discriminator as a module taking two classes of inputs: genuine and fake. Suppose the discriminator outputs <span class="math-container">$\hat{y} \in [0, 1]$</span> for the input <span class="math-container">$x'$</span> instead of actual label <span class="math-container">$y \in \{0, 1\}$</span> then the binary cross-entropy loss is given by</p>
  <p><span class="math-container">$$L_{CE} = y \log \hat{y} + (1-y) \log(1-\hat{y})$$</span>
  <span class="math-container">$$\implies L_{CE} = y \log D(x') + (1-y) \log(1 - D(x'))$$</span></p>
  <p>Suppose the input <span class="math-container">$x'$</span> is a genuine one <span class="math-container">$x$</span> then <span class="math-container">$y = 1$</span> and</p>
  <p><span class="math-container">$$\implies L_{CE} = \log D(x)$$</span></p>
  <p>Suppose the input <span class="math-container">$x'$</span> is a fake one <span class="math-container">$G(z)$</span> then <span class="math-container">$y = 0$</span> and</p>
  <p><span class="math-container">$$\implies L_{CE} = \log (1-D(G(z)))$$</span></p>
  <p>Since the labels are clear from the input of the discriminator <span class="math-container">$D$</span>, we can write the binary cross-entropy loss for <span class="math-container">$2m$</span> samples <span class="math-container">$\{x_1, x_2, x_3, \cdots, x_m, z_1, z_2, z_3, \cdots, z_m\}$</span> as</p>
  <p><span class="math-container">$$\implies L_{CE}^{2m} = \dfrac{1}{2m} \sum\limits_{i = 1}^{m} \log D(x_i) +  \sum\limits_{i = 1}^{m} \log (1-D(G(z_i)))$$</span></p>
  <p>Later, we need to perform some mathematical analysis, which I am not sure about whether it is due to law of large numbers or some other <a href="https://ai.stackexchange.com/questions/29965/does-average-loss-function-in-gan-training-is-just-an-approximation-of-value-fun">1</a>, <a href="https://ai.stackexchange.com/questions/29953/what-are-the-iid-random-variables-for-a-dataset-in-the-gan-framework">2</a> we equate the mean to the actual expectations on probability distribution and hence</p>
  <p><span class="math-container">$$ L_{CE}^{2m} = \dfrac{1}{2} \sum\limits_{i = 1}^{m} \dfrac{1}{m} \log D(x_i) +  \sum\limits_{i = 1}^{m} \dfrac{1}{m} \log (1-D(G(z_i)))$$</span></p>
  <p><span class="math-container">$$ = \dfrac{1}{2} {\LARGE(} \mathbb{E}_{x ∼ P_{data}}[\log D(x)] +  \mathbb{E}_{z ∼ p_z}[log (1 - D(G(z)))] {\LARGE)}$$</span></p>
  <p>Since removing <span class="math-container">$\dfrac{1}{2}$</span> <a href="https://math.stackexchange.com/a/884903/98214">does not matter</a> while optimizing the loss function, the final loss function is given by</p>
  <p><span class="math-container">$$  L_{D} =  \mathbb{E}_{x ∼ P_{data}}[\log D(x)] +\mathbb{E}_{z ∼ p_z}[log (1 - D(G(z)))]$$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-03T08:23:06.730Z
Answers.Text: You are wrong about the binary cross-entropy for GANs. While this is true for the common classification problem, in the GAN we have both real and fake data at the same time. So both labels are correct and equal to 1 (true for real and true for fake). I also wrote it in my [answer](https://ai.stackexchange.com/a/29973/12841).
Answers.UserDisplayName: Aray Karjauv
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-03T10:50:56.200Z
Answers.Text: @ArayKarjauv I don't think so. Discriminator has only two labels  1 for genuine $x$ and 0 or fake $G(z)$. If you are sure enough, please point the exact step where the derivation is wrong.
Answers.UserDisplayName: hanugm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-03T14:13:09.703Z
Answers.Text: Your conclusion that the loss becomes $L_{CE} = \log D(x)$ when $y = 1$ and $L_{CE} = \log (1-D(G(z)))$ when $y = 0$  is wrong. The intuition is that these 2 terms have different labels. The first one is $y_{real}$, whereas the second is $y_{fake}$, which always equal to 1, hence, the loss is always $\implies L(D, G) =  \log D(x_i) +  \log (1-D(G(z_i)))$.
Answers.UserDisplayName: Aray Karjauv
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-03T14:17:03.650Z
Answers.Text: You can think of it as if the loss were a combination of 2 cross-entropy loss functions: one for real data and one for fake. But since the labels there would be constant, we simplify it.
Answers.UserDisplayName: Aray Karjauv
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-05T00:45:55.507Z
Answers.Text: @ArayKarjauv I can't figure out where can it be wrong. In case of your analysis, it seems to me that there are four labels and in the analysis I did, there are only two labels.
Answers.UserDisplayName: hanugm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-03T01:48:00.720Z
Answers.LastActivityDate: 2021-09-06T21:59:41.690Z
Answers.LastEditDate: 2021-09-06T21:59:41.690Z
Answers.UserDisplayName: hanugm
Body: <p>To train the discriminator network in GANs we set the label for the true samples as <span class="math-container">$1$</span> and <span class="math-container">$0$</span> for fake ones. Then we use binary cross-entropy loss for training.</p>
  <p>Since we set the label <span class="math-container">$1$</span> for true samples that means <span class="math-container">$p_{data}(x) = 1$</span> and now binary cross-entropy loss is:
  <span class="math-container">$$L_1 = \sum_{i=1}^{N} P_{data}(x_i)log(D(x)) + (1-P_{data}(x_i))log(1-D(x))$$</span>
  <span class="math-container">$$L_1 = \sum_{i=1}^{N} P_{data}(x_i)log(D(x))$$</span>
  <span class="math-container">$$L_1 = E_{x \sim P_{data}(x)}[log(D(x))]$$</span></p>
  <p>For the second part, since we set the label <span class="math-container">$0$</span> for fake samples that means <span class="math-container">$p_{z}(z) = 0$</span> and now binary cross-entropy loss is:
  <span class="math-container">$$L_2 = \sum_{i=1}^{N} P_{z}(z_i)log(D_{G}(z)) + (1-P_{z}(z_i))log(1-D_{G}(z))$$</span>
  <span class="math-container">$$L_2 = \sum_{i=1}^{N} 1-P_{z}(z_i)log(1-D_{G}(z))$$</span>
  <span class="math-container">$$L_2 = E_{z \sim \bar{P_{z}(z)}}[log(1-D_{G}(z))]$$</span>
  Now we combine those two losses and get:
  <span class="math-container">$$L_D = E_{x \sim P_{data}(x)}[log(D(x))] + E_{z \sim \bar{P_{z}(z)}}[log(1-D_{G}(z))]$$</span>
  When I was reading about <a href="https://arxiv.org/pdf/1406.2661.pdf" rel="nofollow noreferrer">GANs</a> I saw that the loss function for discriminator is defined as:
  <span class="math-container">$$L_D = E_{x \sim P_{data}(x)}[log(D(x))] + E_{z \sim P_{z}(z)}[log(1-D_{G}(z))]$$</span>
  Should not it be <span class="math-container">$E_{z \sim \bar{P_{z}(z)}}$</span> instead of <span class="math-container">$E_{z \sim P_{z}(z)}$</span> ?</p>
  
Tags: <deep-learning><objective-functions><generative-adversarial-networks><generative-model><discriminator>
Title: How to define loss function for Discriminator in GANs?
UserDisplayName: Swakshar Deb
ThreadText: # How to define loss function for Discriminator in GANs?
  
  (Asked by: username_0 on 2020-10-16)
  
  To train the discriminator network in GANs we set the label for the true samples as $1$ and $0$ for fake ones. Then we use binary cross-entropy loss for training.
  
  Since we set the label $1$ for true samples that means $p\_{data}(x) = 1$ and now binary cross-entropy loss is:
  $$L\_1 = \sum\_{i=1}^{N} P\_{data}(x\_i)log(D(x)) + (1-P\_{data}(x\_i))log(1-D(x))$$
  $$L\_1 = \sum\_{i=1}^{N} P\_{data}(x\_i)log(D(x))$$
  $$L\_1 = E\_{x \sim P\_{data}(x)}[log(D(x))]$$
  
  For the second part, since we set the label $0$ for fake samples that means $p\_{z}(z) = 0$ and now binary cross-entropy loss is:
  $$L\_2 = \sum\_{i=1}^{N} P\_{z}(z\_i)log(D\_{G}(z)) + (1-P\_{z}(z\_i))log(1-D\_{G}(z))$$
  $$L\_2 = \sum\_{i=1}^{N} 1-P\_{z}(z\_i)log(1-D\_{G}(z))$$
  $$L\_2 = E\_{z \sim \bar{P\_{z}(z)}}[log(1-D\_{G}(z))]$$
  Now we combine those two losses and get:
  $$L\_D = E\_{x \sim P\_{data}(x)}[log(D(x))] + E\_{z \sim \bar{P\_{z}(z)}}[log(1-D\_{G}(z))]$$
  When I was reading about [GANs](https://arxiv.org/pdf/1406.2661.pdf) I saw that the loss function for discriminator is defined as:
  $$L\_D = E\_{x \sim P\_{data}(x)}[log(D(x))] + E\_{z \sim P\_{z}(z)}[log(1-D\_{G}(z))]$$
  Should not it be $E\_{z \sim \bar{P\_{z}(z)}}$ instead of $E\_{z \sim P\_{z}(z)}$ ?
  
  **Tags:** deep-learning, objective-functions, generative-adversarial-networks, generative-model, discriminator
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-08-03. Score: 1
  
  I doubt whether your derivation is correct. Your are trying to apply binary-cross entropy for data on each label separately, which is not the correct way to do.
  
  The procedure for calculating binary cross entropy is as follows
  
  1. Pass the input $x$ whose label is $y \in \{0, 1\}$ to your model $M$.
  2. Obtain $\hat{y} \in [0, 1]$ as output of your model $M$ instead of actual label $y$.
  3. Calculate binary cross-entropy loss using the equation
  
  $$L\_{CE} = y \log \hat{y} + (1-y) \log (1 - \hat{y})$$
  
  It is true that there are two types of inputs to a discriminator: genuine and fake. Genuine data is labelled by 1 and fake data is labelled by 0. Use the variable $x'$ to represent the input to the discriminator module $D$. If the input $x'$ is genuine then its label is 1 and if your input $x'$ is fake then its label is 0. Note that it is better to avoid the unnecessary details regarding the generator or noise vector while formulating the binary cross-entropy loss of discriminator. Just see discriminator as a module taking two classes of inputs: genuine and fake. Suppose the discriminator outputs $\hat{y} \in [0, 1]$ for the input $x'$ instead of actual label $y \in \{0, 1\}$ then the binary cross-entropy loss is given by
  
  $$L\_{CE} = y \log \hat{y} + (1-y) \log(1-\hat{y})$$
  $$\implies L\_{CE} = y \log D(x') + (1-y) \log(1 - D(x'))$$
  
  Suppose the input $x'$ is a genuine one $x$ then $y = 1$ and
  
  $$\implies L\_{CE} = \log D(x)$$
  
  Suppose the input $x'$ is a fake one $G(z)$ then $y = 0$ and
  
  $$\implies L\_{CE} = \log (1-D(G(z)))$$
  
  Since the labels are clear from the input of the discriminator $D$, we can write the binary cross-entropy loss for $2m$ samples $\{x\_1, x\_2, x\_3, \cdots, x\_m, z\_1, z\_2, z\_3, \cdots, z\_m\}$ as
  
  $$\implies L\_{CE}^{2m} = \dfrac{1}{2m} \sum\limits\_{i = 1}^{m} \log D(x\_i) + \sum\limits\_{i = 1}^{m} \log (1-D(G(z\_i)))$$
  
  Later, we need to perform some mathematical analysis, which I am not sure about whether it is due to law of large numbers or some other [1](https://ai.stackexchange.com/questions/29965/does-average-loss-function-in-gan-training-is-just-an-approximation-of-value-fun), [2](https://ai.stackexchange.com/questions/29953/what-are-the-iid-random-variables-for-a-dataset-in-the-gan-framework) we equate the mean to the actual expectations on probability distribution and hence
  
  $$ L\_{CE}^{2m} = \dfrac{1}{2} \sum\limits\_{i = 1}^{m} \dfrac{1}{m} \log D(x\_i) + \sum\limits\_{i = 1}^{m} \dfrac{1}{m} \log (1-D(G(z\_i)))$$
  
  $$ = \dfrac{1}{2} {\LARGE(} \mathbb{E}\_{x ∼ P\_{data}}[\log D(x)] + \mathbb{E}\_{z ∼ p\_z}[log (1 - D(G(z)))] {\LARGE)}$$
  
  Since removing $\dfrac{1}{2}$ [does not matter](https://math.stackexchange.com/a/884903/98214) while optimizing the loss function, the final loss function is given by
  
  $$ L\_{D} = \mathbb{E}\_{x ∼ P\_{data}}[\log D(x)] +\mathbb{E}\_{z ∼ p\_z}[log (1 - D(G(z)))]$$
  
  > **Comments:**
  > **username_2**: You are wrong about the binary cross-entropy for GANs. While this is true for the common classification problem, in the GAN we have both real and fake data at the same time. So both labels are correct and equal to 1 (true for real and true for fake). I also wrote it in my [answer](https://ai.stackexchange.com/a/29973/12841).  
  > **username_1**: @ArayKarjauv I don't think so. Discriminator has only two labels  1 for genuine $x$ and 0 or fake $G(z)$. If you are sure enough, please point the exact step where the derivation is wrong.  
  > **username_2**: Your conclusion that the loss becomes $L_{CE} = \log D(x)$ when $y = 1$ and $L_{CE} = \log (1-D(G(z)))$ when $y = 0$  is wrong. The intuition is that these 2 terms have different labels. The first one is $y_{real}$, whereas the second is $y_{fake}$, which always equal to 1, hence, the loss is always $\implies L(D, G) =  \log D(x_i) +  \log (1-D(G(z_i)))$.  
  > **username_2**: You can think of it as if the loss were a combination of 2 cross-entropy loss functions: one for real data and one for fake. But since the labels there would be constant, we simplify it.  
  > **username_1**: @ArayKarjauv I can't figure out where can it be wrong. In case of your analysis, it seems to me that there are four labels and in the analysis I did, there are only two labels.  
  
  ---
  
  