Answers.Body: <p><span class="math-container">$\textbf{Remark.}$</span> I'd leave this as a comment if I could.</p>
  <p>Regarding notation (which I believe may be the cause of your issue here), the loss function is better written as
  <span class="math-container">\begin{align*}
  \operatorname{Loss} &amp;= \frac{1}{m}\sum_{i=1}^m \left(\log D\big(x^{(i)}\big) + \log\Big(1-D\big(G\big(z^{(i)}\big)\big)\right)\\
  &amp;\approx \mathbb{E}_x[\log D(x)] + \mathbb{E}_z[\log(1-D(G(z)))],
  \end{align*}</span>
  where the noise vectors, <span class="math-container">$z$</span>, come from a suitable distribution, and <span class="math-container">$G(z)$</span> denotes the output of the generator; the <span class="math-container">$\approx$</span> symbol here implicitly assumes that the appropriate form of the Law of Large Numbers (LLN) applies.</p>
  <p>Most importantly, the dependence on G is not trivial (for instance, what if <span class="math-container">$G$</span> never learns and always produces the same output?).</p>
  <p>Also, the expectations should depend on their respective distributions, even when using LLN. For example, think of how you calculate the expectation of a discrete random variable.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-31T23:46:48.113Z
Answers.Text: Thanks for the contribution. However, output of G(z) is another probability distribution over x i.e. p_g(x). So, we can write that equation like I wrote with confidence. Also, in the original paper It is also written like that (https://arxiv.org/pdf/1406.2661.pdf at part 4.1). Also, when we convert two expectation into sigma form, they have same sigma in the same domain (x), so I thought it is okay.
Answers.UserDisplayName: Enes
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-01T14:17:49.047Z
Answers.Text: That's fine, but you're assuming that p_r(x) and p_g(x) come from the same distribution just because they have the same domain. 
  
  For example, apply your original argument to some random variable X and any function f. It would imply that X and f(X) have the same mean, which clearly isn't true in general.
Answers.UserDisplayName: Scott Kirila
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-01T16:35:48.667Z
Answers.Text: Also, When we focus on distribuitons individually, $ CE(p_r(x), D(x))= \lim_{m \to \infty} \frac{1}{m} CE(1, D(x)) $. Is it valid equation?
Answers.UserDisplayName: Enes
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-31T23:35:21.190Z
Answers.LastActivityDate: 2020-12-31T23:35:21.190Z
Answers.UserDisplayName: Scott Kirila
Answers.Body: <p>I guess the issue is you lost track of where the samples came from and since you requested a math explanation I'll try to go step by step using my notation and without checking other material to avoid being biased by how other authors present it</p>
  <p>So we start from</p>
  <p><span class="math-container">$$ L(D,G) = E_{x \sim p_{r}(x)} \log(D(x)) + E_{x \sim p_{g}(x)}\log(1 - D(x)) $$</span></p>
  <p>then you apply the definition of <span class="math-container">$E_{\cdot}(\cdot)$</span> operator in the continuous case</p>
  <p><span class="math-container">$$ L(D,G) = \int_{x} \log(D(x)) p_{r}(x)dx + \int_{x}\log(1 - D(x))p_{g}(x)dx $$</span></p>
  <p>then you Monte Carlo sample it to approximate it</p>
  <p><span class="math-container">$$ L(D,G) = \frac{1}{n} \sum_{i=1}^{n} \log(D(x_{i}^{(r)}))  + \frac{1}{m} \sum_{j=1}^{m}\log(1 - D(x_{j}^{(g)})) $$</span></p>
  <p>As you can see here I have kept the samples from the 2 distributions separated and used a notation that allows to track their origin so now you can use the right label in the Cross Entropy</p>
  <p><span class="math-container">$$ L(D,G) = \frac{1}{n} \sum_{i=1}^{n} L_{ce}(1, D(x_{i}^{(r)}))  + \frac{1}{m} \sum_{j=1}^{m} L_{ce}(0, D(x_{j}^{(g)})) $$</span></p>
  <p>But you could also have decided to merge the 2 integrals before to have</p>
  <p><span class="math-container">$$ L(D,G) = \int_{x} \left( \log(D(x)) p_{r}(x) + \log(1 - D(x))p_{g}(x) \right) dx $$</span></p>
  <p>which is mathematically legit operation, however the issue is when you try to
  discretize this with Monte Carlo sampling.</p>
  <p>You can't just replace the integral with one sum since you are Monte Carlo sampling and here, contrary to what we have done above, you do not have 1 distribution per integral to sample but in the same integral you have 2 distributions and for each sample you have to say what distribution it comes from which is where the issue is in your notation since you lost track of this information and it seems all the samples come from one distribution</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-01T12:47:24.437Z
Answers.Text: Thanks a lot for the answer. Could you suggest me any resource or reading about this aspect of Monte Carlo Sampling (or in general)
Answers.UserDisplayName: Enes
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-01T16:35:10.080Z
Answers.Text: Also, When we focus on distribuitons individually, $ CE(p_r(x), D(x))= \lim_{m \to \infty} \frac{1}{m} CE(1, D(x)) $. Is it valid equation?
Answers.UserDisplayName: Enes
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-01T19:39:35.210Z
Answers.Text: I guess what you would like to do is 
  
  $$ \lim_{m \rightarrow \infty} \frac{1}{m} \sum_{i=1}^{m} CE(1, D(x_{i}^{(r)})) $$
  
  but this is not approximating what you have on the left side since the above assumes this is a real sample (see the $r$ at the top) while on the left it is a generic sample, meaning you do not know what distribution it comes from, and this is reflected in first argument of the cross entropy: one is the real sample distribution and the other is a Dirac Delta since it is by construction real
Answers.UserDisplayName: Nicola Bernini
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-01T09:44:58.490Z
Answers.LastActivityDate: 2021-01-03T23:23:42.923Z
Answers.LastEditDate: 2021-01-03T23:23:42.923Z
Answers.UserDisplayName: Nicola Bernini
Body: <p>I was pondering on the loss function of GAN, and the following thing turned out</p>
  <p><span class="math-container">\begin{aligned}
   L(D, G) 
  &amp; = \mathbb{E}_{x \sim p_{r}(x)} [\log D(x)] + \mathbb{E}_{x \sim p_g(x)} [\log(1 - D(x)] \\
  &amp; = \int_x \bigg( p_{r}(x) \log(D(x)) + p_g (x) \log(1 - D(x)) \bigg) dx \\ 
  &amp; =-\left[CE(p_r(x), D(x))+CE(p_g(x), 1-D(x)) \right] \\
  \end{aligned}</span>
  Where CE stands for cross-entropy. Then, by using law of large numbers:
  <span class="math-container">\begin{aligned}
  L(D, G) 
  &amp; = \mathbb{E}_{x \sim p_{r}(x)} [\log D(x)] + \mathbb{E}_{x \sim p_g(x)} [\log(1 - D(x)] \\
  &amp; =\lim_{m\to \infty}\frac{1}{m}\sum_{i=1}^{m}\left[1\cdot \log(D(x^{(i)}))+1\cdot \log(1-D(x^{(i)}))\right]\\
  &amp; =- \lim_{m \to \infty} \frac{1}{m}\sum_{i=1}^{m} \left[CE(1, D(x))+CE(0, D(x))\right]
  \end{aligned}</span></p>
  <p>As you can see, I got a very strange result. This should be wrong intuitively because in the last equation first part is for real samples, and the second is for generated samples. However, I am curious about where are the mistakes?</p>
  <p>(Please explain with math).</p>
  
Tags: <deep-learning><objective-functions><generative-adversarial-networks><generative-model><cross-entropy>
Title: Where is the mistake in my derivation of the GAN loss function?
UserDisplayName: Enes
ThreadText: # Where is the mistake in my derivation of the GAN loss function?
  
  (Asked by: username_0 on 2020-12-31)
  
  I was pondering on the loss function of GAN, and the following thing turned out
  
  \begin{aligned}
  L(D, G)
  & = \mathbb{E}\_{x \sim p\_{r}(x)} [\log D(x)] + \mathbb{E}\_{x \sim p\_g(x)} [\log(1 - D(x)] \\
  & = \int\_x \bigg( p\_{r}(x) \log(D(x)) + p\_g (x) \log(1 - D(x)) \bigg) dx \\
  & =-\left[CE(p\_r(x), D(x))+CE(p\_g(x), 1-D(x)) \right] \\
  \end{aligned}
  Where CE stands for cross-entropy. Then, by using law of large numbers:
  \begin{aligned}
  L(D, G)
  & = \mathbb{E}\_{x \sim p\_{r}(x)} [\log D(x)] + \mathbb{E}\_{x \sim p\_g(x)} [\log(1 - D(x)] \\
  & =\lim\_{m\to \infty}\frac{1}{m}\sum\_{i=1}^{m}\left[1\cdot \log(D(x^{(i)}))+1\cdot \log(1-D(x^{(i)}))\right]\\
  & =- \lim\_{m \to \infty} \frac{1}{m}\sum\_{i=1}^{m} \left[CE(1, D(x))+CE(0, D(x))\right]
  \end{aligned}
  
  As you can see, I got a very strange result. This should be wrong intuitively because in the last equation first part is for real samples, and the second is for generated samples. However, I am curious about where are the mistakes?
  
  (Please explain with math).
  
  **Tags:** deep-learning, objective-functions, generative-adversarial-networks, generative-model, cross-entropy
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2021-01-01. Score: 4 | Accepted answer
  
  I guess the issue is you lost track of where the samples came from and since you requested a math explanation I'll try to go step by step using my notation and without checking other material to avoid being biased by how other authors present it
  
  So we start from
  
  $$ L(D,G) = E\_{x \sim p\_{r}(x)} \log(D(x)) + E\_{x \sim p\_{g}(x)}\log(1 - D(x)) $$
  
  then you apply the definition of $E\_{\cdot}(\cdot)$ operator in the continuous case
  
  $$ L(D,G) = \int\_{x} \log(D(x)) p\_{r}(x)dx + \int\_{x}\log(1 - D(x))p\_{g}(x)dx $$
  
  then you Monte Carlo sample it to approximate it
  
  $$ L(D,G) = \frac{1}{n} \sum\_{i=1}^{n} \log(D(x\_{i}^{(r)})) + \frac{1}{m} \sum\_{j=1}^{m}\log(1 - D(x\_{j}^{(g)})) $$
  
  As you can see here I have kept the samples from the 2 distributions separated and used a notation that allows to track their origin so now you can use the right label in the Cross Entropy
  
  $$ L(D,G) = \frac{1}{n} \sum\_{i=1}^{n} L\_{ce}(1, D(x\_{i}^{(r)})) + \frac{1}{m} \sum\_{j=1}^{m} L\_{ce}(0, D(x\_{j}^{(g)})) $$
  
  But you could also have decided to merge the 2 integrals before to have
  
  $$ L(D,G) = \int\_{x} \left( \log(D(x)) p\_{r}(x) + \log(1 - D(x))p\_{g}(x) \right) dx $$
  
  which is mathematically legit operation, however the issue is when you try to
  discretize this with Monte Carlo sampling.
  
  You can't just replace the integral with one sum since you are Monte Carlo sampling and here, contrary to what we have done above, you do not have 1 distribution per integral to sample but in the same integral you have 2 distributions and for each sample you have to say what distribution it comes from which is where the issue is in your notation since you lost track of this information and it seems all the samples come from one distribution
  
  > **Comments:**
  > **username_0**: Thanks a lot for the answer. Could you suggest me any resource or reading about this aspect of Monte Carlo Sampling (or in general)  
  > **username_0**: Also, When we focus on distribuitons individually, $ CE(p_r(x), D(x))= \lim_{m \to \infty} \frac{1}{m} CE(1, D(x)) $. Is it valid equation?  
  > **username_2**: I guess what you would like to do is   $$ \lim_{m \rightarrow \infty} \frac{1}{m} \sum_{i=1}^{m} CE(1, D(x_{i}^{(r)})) $$  but this is not approximating what you have on the left side since the above assumes this is a real sample (see the $r$ at the top) while on the left it is a generic sample, meaning you do not know what distribution it comes from, and this is reflected in first argument of the cross entropy: one is the real sample distribution and the other is a Dirac Delta since it is by construction real  
  
  ---
  
  ### Answer by username_1 on 2020-12-31. Score: 1
  
  $\textbf{Remark.}$ I'd leave this as a comment if I could.
  
  Regarding notation (which I believe may be the cause of your issue here), the loss function is better written as
  \begin{align\*}
  \operatorname{Loss} &= \frac{1}{m}\sum\_{i=1}^m \left(\log D\big(x^{(i)}\big) + \log\Big(1-D\big(G\big(z^{(i)}\big)\big)\right)\\
  &\approx \mathbb{E}\_x[\log D(x)] + \mathbb{E}\_z[\log(1-D(G(z)))],
  \end{align\*}
  where the noise vectors, $z$, come from a suitable distribution, and $G(z)$ denotes the output of the generator; the $\approx$ symbol here implicitly assumes that the appropriate form of the Law of Large Numbers (LLN) applies.
  
  Most importantly, the dependence on G is not trivial (for instance, what if $G$ never learns and always produces the same output?).
  
  Also, the expectations should depend on their respective distributions, even when using LLN. For example, think of how you calculate the expectation of a discrete random variable.
  
  > **Comments:**
  > **username_0**: Thanks for the contribution. However, output of G(z) is another probability distribution over x i.e. p_g(x). So, we can write that equation like I wrote with confidence. Also, in the original paper It is also written like that (https://arxiv.org/pdf/1406.2661.pdf at part 4.1). Also, when we convert two expectation into sigma form, they have same sigma in the same domain (x), so I thought it is okay.  
  > **username_1**: That's fine, but you're assuming that p_r(x) and p_g(x) come from the same distribution just because they have the same domain.   For example, apply your original argument to some random variable X and any function f. It would imply that X and f(X) have the same mean, which clearly isn't true in general.  
  > **username_0**: Also, When we focus on distribuitons individually, $ CE(p_r(x), D(x))= \lim_{m \to \infty} \frac{1}{m} CE(1, D(x)) $. Is it valid equation?  
  
  ---
  
  