Answers.Body: <p><strong>Expected value</strong> can be thought of as a weighted average of outcomes. Thus, expectation and mean are the same thing, if each outcome has the same probability (which is <span class="math-container">$\frac{1}{m}$</span>), so we can replace it with a sum divided by <span class="math-container">$m$</span>. We can rewrite the equation:
  <span class="math-container">$$\min_G \max_DV(D, G) = \mathbb{E}_{x ∼ P_{data}}[\log D(x)] +  \mathbb{E}_{z ∼ p_z}[log (1 - D(G(z)))]$$</span></p>
  <p>First, we sample minibatch of size <span class="math-container">$m$</span> for <span class="math-container">$\boldsymbol{x} \sim P_{data}$</span> and <span class="math-container">$\boldsymbol{z} \sim \mathcal{N(0, 1)}$</span>. Now we can replace the expectation with the sum:</p>
  <p><span class="math-container">$$
  \begin{align*}
  \min_G \max_DV(D, G) &amp;= \sum_{i=1}^{m}\left[p(\boldsymbol{x}^{(i)})\log D(\boldsymbol{x}^{(i)})\right] +  \sum_{i=1}^{m}\left[p(\boldsymbol{z}^{(i)})log (1 - D(G(\boldsymbol{z}^{(i)})))\right] \\
  &amp;= \sum_{i=1}^{m}\left[\frac{1}{m}\log D(\boldsymbol{x}^{(i)})\right] + \sum_{i=1}^{m}[\frac{1}{m}log (1 - D(G(\boldsymbol{z}^{(i)})))]\\
   &amp;=\frac{1}{m}\sum_{i=1}^{m}\left[\log D(\boldsymbol{x}^{(i)}) + log (1 - D(G(\boldsymbol{z}^{(i)})))\right]
  \end{align*}
  $$</span></p>
  <hr />
  <p><strong>Binary cross entropy</strong> defined as follows:</p>
  <p><span class="math-container">$$H(p, q) = \operatorname{E}_p[-\log q] = H(p) + D_{\mathrm{KL}}(p \| q)=-\sum_x p(x)\log q(x)$$</span></p>
  <p>Since we have a binary classification problem (fake/real), we can define <span class="math-container">$p \in \{y,1-y\}$</span> and <span class="math-container">$q \in \{\hat{y}, 1-\hat{y}\}$</span> and rewriting coros entropy as follows:</p>
  <p><span class="math-container">$$H(p, q)=-\sum_x p_x \log q_x =-y\log \hat{y}-(1-y)\log (1-\hat{y})$$</span></p>
  <p>which is nothing but logistic loss. Since we know the source of our data (either real or fake), we can replace labels <span class="math-container">$y$</span> for real and fake with 1. We then get:
  <span class="math-container">$$\min_G\max_D L =  \frac{1}{m} \sum_{i=1}^{m}\left[1\cdot\log D\left(\boldsymbol{x}^{(i)}\right)+1\cdot\log \left(1-D\left(G\left(\boldsymbol{z}^{(i)}\right)\right)\right)\right]
  $$</span></p>
  <p>This is the original loss. The first term in the equation gets always real images, while the second gets only generated. Hence, both terms have corresponding true labels. Read this <a href="https://danieltakeshi.github.io/2017/03/05/understanding-generative-adversarial-networks/" rel="nofollow noreferrer">article</a> for more details.</p>
  <p>Since the first term does not depend on <span class="math-container">$G$</span>, we can rewrite it as follows:</p>
  <p><span class="math-container">$$\max L(D) =  \frac{1}{m} \sum_{i=1}^{m}\left[\log D\left(\boldsymbol{x}^{(i)}\right)+\log \left(1-D\left(G\left(\boldsymbol{z}^{(i)}\right)\right)\right)\right]$$</span></p>
  <p><span class="math-container">$$\min L(G) =  \frac{1}{m} \sum_{i=1}^{m}\left[\log \left(1-D\left(G\left(\boldsymbol{z}^{(i)}\right)\right)\right)\right]$$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-02T23:09:22.137Z
Answers.Text: My doubt is in fixing up the probabilities of noise, generated and real samples as $\dfrac{1}{m}$, where $m$ is just a hyperparameter we are selecting and does not depends on actual probability. So, how can we take that?
Answers.UserDisplayName: hanugm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-03T01:15:12.280Z
Answers.Text: I guess I [answered](https://ai.stackexchange.com/questions/29953/what-are-the-iid-random-variables-for-a-dataset-in-the-gan-framework/29958#29958) the question about $m$ - we assume each image has the same probability (we can also sample images with certain probabilities). This is not so important. We could have made a batch size 1, but as far as I know, batch optimization is less prone to getting stuck in local minima. As for the noise, it is only used to generate images, so we can replace $G(z)$ with $\hat{x} \sim P_g$
Answers.UserDisplayName: Aray Karjauv
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-03T02:19:23.403Z
Answers.Text: yeah, I understood. But I am not sure whether considering our own probability is theoretically correct or nor. So, I am still in confusion.
Answers.UserDisplayName: hanugm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-03T08:45:26.407Z
Answers.Text: @hanugm I don't think the mean is related to the probabilities of the data in this case. It has more to do with the importance of the gradient in the batch. The gradient for all data points is equally important. You can update your model using one data point at a time, so you get rid of the mean. The likelihood of the data is encoded in your dataset. If your dataset is unbalanced, you can sample a specific class with a certain probability, for example, using [WeightedRandomSampler](https://pytorch.org/docs/stable/data.html) in PyTorch.
Answers.UserDisplayName: Aray Karjauv
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-02T12:24:23.573Z
Answers.LastActivityDate: 2021-08-02T12:39:12.760Z
Answers.LastEditDate: 2021-08-02T12:39:12.760Z
Answers.UserDisplayName: Aray Karjauv
Body: <p>The value function on which convergence has been proved by the <a href="https://papers.nips.cc/paper/2014/file/5ca3e9b122f61f8f06494c97b1afccf3-Paper.pdf" rel="nofollow noreferrer">original paper of GAN</a> is</p>
  <p><span class="math-container">$$\min_G \max_DV(D, G) = \mathbb{E}_{x ∼ P_{data}}[\log D(x)] +  \mathbb{E}_{z ∼ p_z}[log (1 - D(G(z)))]$$</span></p>
  <p>and the loss function used in training are</p>
  <p><span class="math-container">$$\max L(D) =  \frac{1}{m} \sum_{i=1}^{m}\left[\log D\left(\boldsymbol{x}^{(i)}\right)+\log \left(1-D\left(G\left(\boldsymbol{z}^{(i)}\right)\right)\right)\right]$$</span></p>
  <p><span class="math-container">$$\min L(G) =  \frac{1}{m} \sum_{i=1}^{m}\left[\log \left(1-D\left(G\left(\boldsymbol{z}^{(i)}\right)\right)\right)\right]$$</span></p>
  <p>where <span class="math-container">$\{z^{(1)}, z^{(2)}, z^{(3)}, \cdots, z^{(m)}\}$</span> and  <span class="math-container">$\{x^{(1)}, x^{(2)}, x^{(3)}, \cdots, x^{(m)}\}$</span>  ate the noise samples and data samples for a mini-batch respectively.</p>
  <p>I found after analyzing some questions <a href="https://ai.stackexchange.com/questions/29953/what-are-the-iid-random-variables-for-a-dataset-in-the-gan-framework">1</a>, <a href="https://ai.stackexchange.com/questions/23161/why-is-the-mean-used-to-compute-the-expectation-in-the-gan-loss">2</a> on our main site that the loss function used for training is just an approximation of the value function and are not same in formal sense.</p>
  <p>Is it true? If yes, what is the reason behind the disparity? Is the loss function used for implementation also ensures convergence?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-02T10:56:38.183Z
Comments.Text: What do you mean by "an approximation of the value function and are not same in formal sense"? The original loss is nothing more than the BCE loss. The generator's loss is more complex: D plays the role of some kind of dynamic loss function.
Comments.UserDisplayName: Aray Karjauv
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-02T11:02:43.047Z
Comments.Text: @ArayKarjauv I mean, they both are not same but latter is inspired on former. What do you mean by original loss function? Is it value function?
Comments.UserDisplayName: hanugm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-02T11:10:12.530Z
Comments.Text: Do you mean the difference between the loss functions for D and G?
Comments.UserDisplayName: Aray Karjauv
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-02T11:11:11.850Z
Comments.Text: @ArayKarjauv No, the difference between the actual value function and the loss function used for implementation.
Comments.UserDisplayName: hanugm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-02T11:14:20.033Z
Comments.Text: They are identical. The notation is slightly different. The only difference is the loss for G.
Comments.UserDisplayName: Aray Karjauv
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-02T11:15:24.937Z
Comments.Text: @ArayKarjauv How can be the same? The value function is using the actual probabilities and in implementation, we are using just 1/m!
Comments.UserDisplayName: hanugm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-02T11:16:31.730Z
Comments.Text: @ArayKarjauv Log part is the same. But when comes to the probability of samples, they are different right?
Comments.UserDisplayName: hanugm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-02T11:18:47.967Z
Comments.Text: @ArayKarjauv Are you saying that both are BCE? If yes then I may be facing difficulty in deriving the loss functions used for implementation from the theoretical value function used...
Comments.UserDisplayName: hanugm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-02T13:45:14.080Z
Comments.Text: I decided to write a complete answer. Feel free to leave your feedback.
Comments.UserDisplayName: Aray Karjauv
Tags: <comparison><training><objective-functions><generative-adversarial-networks>
Title: Does average loss function in GAN training is just an approximation of value function and does not ensure convergence of generator and discriminator?
UserDisplayName: hanugm
ThreadText: # Does average loss function in GAN training is just an approximation of value function and does not ensure convergence of generator and discriminator?
  
  (Asked by: username_0 on 2021-08-02)
  
  The value function on which convergence has been proved by the [original paper of GAN](https://papers.nips.cc/paper/2014/file/5ca3e9b122f61f8f06494c97b1afccf3-Paper.pdf) is
  
  $$\min\_G \max\_DV(D, G) = \mathbb{E}\_{x ∼ P\_{data}}[\log D(x)] + \mathbb{E}\_{z ∼ p\_z}[log (1 - D(G(z)))]$$
  
  and the loss function used in training are
  
  $$\max L(D) = \frac{1}{m} \sum\_{i=1}^{m}\left[\log D\left(\boldsymbol{x}^{(i)}\right)+\log \left(1-D\left(G\left(\boldsymbol{z}^{(i)}\right)\right)\right)\right]$$
  
  $$\min L(G) = \frac{1}{m} \sum\_{i=1}^{m}\left[\log \left(1-D\left(G\left(\boldsymbol{z}^{(i)}\right)\right)\right)\right]$$
  
  where $\{z^{(1)}, z^{(2)}, z^{(3)}, \cdots, z^{(m)}\}$ and $\{x^{(1)}, x^{(2)}, x^{(3)}, \cdots, x^{(m)}\}$ ate the noise samples and data samples for a mini-batch respectively.
  
  I found after analyzing some questions [1](https://ai.stackexchange.com/questions/29953/what-are-the-iid-random-variables-for-a-dataset-in-the-gan-framework), [2](https://ai.stackexchange.com/questions/23161/why-is-the-mean-used-to-compute-the-expectation-in-the-gan-loss) on our main site that the loss function used for training is just an approximation of the value function and are not same in formal sense.
  
  Is it true? If yes, what is the reason behind the disparity? Is the loss function used for implementation also ensures convergence?
  
  **Tags:** comparison, training, objective-functions, generative-adversarial-networks
  > **Comments:**
  > **username_1**: What do you mean by "an approximation of the value function and are not same in formal sense"? The original loss is nothing more than the BCE loss. The generator's loss is more complex: D plays the role of some kind of dynamic loss function.  
  > **username_0**: @ArayKarjauv I mean, they both are not same but latter is inspired on former. What do you mean by original loss function? Is it value function?  
  > **username_1**: Do you mean the difference between the loss functions for D and G?  
  > **username_0**: @ArayKarjauv No, the difference between the actual value function and the loss function used for implementation.  
  > **username_1**: They are identical. The notation is slightly different. The only difference is the loss for G.  
  > **username_0**: @ArayKarjauv How can be the same? The value function is using the actual probabilities and in implementation, we are using just 1/m!  
  > **username_0**: @ArayKarjauv Log part is the same. But when comes to the probability of samples, they are different right?  
  > **username_0**: @ArayKarjauv Are you saying that both are BCE? If yes then I may be facing difficulty in deriving the loss functions used for implementation from the theoretical value function used...  
  > **username_1**: I decided to write a complete answer. Feel free to leave your feedback.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-08-02. Score: 0
  
  **Expected value** can be thought of as a weighted average of outcomes. Thus, expectation and mean are the same thing, if each outcome has the same probability (which is $\frac{1}{m}$), so we can replace it with a sum divided by $m$. We can rewrite the equation:
  $$\min\_G \max\_DV(D, G) = \mathbb{E}\_{x ∼ P\_{data}}[\log D(x)] + \mathbb{E}\_{z ∼ p\_z}[log (1 - D(G(z)))]$$
  
  First, we sample minibatch of size $m$ for $\boldsymbol{x} \sim P\_{data}$ and $\boldsymbol{z} \sim \mathcal{N(0, 1)}$. Now we can replace the expectation with the sum:
  
  $$
  \begin{align\*}
  \min\_G \max\_DV(D, G) &= \sum\_{i=1}^{m}\left[p(\boldsymbol{x}^{(i)})\log D(\boldsymbol{x}^{(i)})\right] + \sum\_{i=1}^{m}\left[p(\boldsymbol{z}^{(i)})log (1 - D(G(\boldsymbol{z}^{(i)})))\right] \\
  &= \sum\_{i=1}^{m}\left[\frac{1}{m}\log D(\boldsymbol{x}^{(i)})\right] + \sum\_{i=1}^{m}[\frac{1}{m}log (1 - D(G(\boldsymbol{z}^{(i)})))]\\
  &=\frac{1}{m}\sum\_{i=1}^{m}\left[\log D(\boldsymbol{x}^{(i)}) + log (1 - D(G(\boldsymbol{z}^{(i)})))\right]
  \end{align\*}
  $$
  
  ---
  
  **Binary cross entropy** defined as follows:
  
  $$H(p, q) = \operatorname{E}\_p[-\log q] = H(p) + D\_{\mathrm{KL}}(p \| q)=-\sum\_x p(x)\log q(x)$$
  
  Since we have a binary classification problem (fake/real), we can define $p \in \{y,1-y\}$ and $q \in \{\hat{y}, 1-\hat{y}\}$ and rewriting coros entropy as follows:
  
  $$H(p, q)=-\sum\_x p\_x \log q\_x =-y\log \hat{y}-(1-y)\log (1-\hat{y})$$
  
  which is nothing but logistic loss. Since we know the source of our data (either real or fake), we can replace labels $y$ for real and fake with 1. We then get:
  $$\min\_G\max\_D L = \frac{1}{m} \sum\_{i=1}^{m}\left[1\cdot\log D\left(\boldsymbol{x}^{(i)}\right)+1\cdot\log \left(1-D\left(G\left(\boldsymbol{z}^{(i)}\right)\right)\right)\right]
  $$
  
  This is the original loss. The first term in the equation gets always real images, while the second gets only generated. Hence, both terms have corresponding true labels. Read this [article](https://danieltakeshi.github.io/2017/03/05/understanding-generative-adversarial-networks/) for more details.
  
  Since the first term does not depend on $G$, we can rewrite it as follows:
  
  $$\max L(D) = \frac{1}{m} \sum\_{i=1}^{m}\left[\log D\left(\boldsymbol{x}^{(i)}\right)+\log \left(1-D\left(G\left(\boldsymbol{z}^{(i)}\right)\right)\right)\right]$$
  
  $$\min L(G) = \frac{1}{m} \sum\_{i=1}^{m}\left[\log \left(1-D\left(G\left(\boldsymbol{z}^{(i)}\right)\right)\right)\right]$$
  
  > **Comments:**
  > **username_0**: My doubt is in fixing up the probabilities of noise, generated and real samples as $\dfrac{1}{m}$, where $m$ is just a hyperparameter we are selecting and does not depends on actual probability. So, how can we take that?  
  > **username_1**: I guess I [answered](https://ai.stackexchange.com/questions/29953/what-are-the-iid-random-variables-for-a-dataset-in-the-gan-framework/29958#29958) the question about $m$ - we assume each image has the same probability (we can also sample images with certain probabilities). This is not so important. We could have made a batch size 1, but as far as I know, batch optimization is less prone to getting stuck in local minima. As for the noise, it is only used to generate images, so we can replace $G(z)$ with $\hat{x} \sim P_g$  
  > **username_0**: yeah, I understood. But I am not sure whether considering our own probability is theoretically correct or nor. So, I am still in confusion.  
  > **username_1**: @hanugm I don't think the mean is related to the probabilities of the data in this case. It has more to do with the importance of the gradient in the batch. The gradient for all data points is equally important. You can update your model using one data point at a time, so you get rid of the mean. The likelihood of the data is encoded in your dataset. If your dataset is unbalanced, you can sample a specific class with a certain probability, for example, using [WeightedRandomSampler](https://pytorch.org/docs/stable/data.html) in PyTorch.  
  
  ---
  
  