Answers.Body: <p><a href="https://en.wikipedia.org/wiki/Independent_and_identically_distributed_random_variables" rel="nofollow noreferrer">Independent and identically distributed</a> random variables share the same probability distribution and each item doesn’t influence or provide insight about the value of the next item you measure. The most common example is a coin toss: as you flip the coin, one outcome does not influence or predict the next one.</p>
  <p>As for a dataset of flowers, we assume that the photos were made independently and all kinds of flowers are evenly represented, which in practice is not true. Each flower won't be represented in the same lighting condition and camera view.</p>
  <p>Each pixel in an image is highly dependent on the neighboring pixels (and sometimes distant ones, e.g., when generating faces as they are usual symmetric) and the main goal of GAN is to discover this dependence by exploring a latent features manifold that represents abstract attributes of the real data, and to find a mapping between these features and output pixels. Of course, if some of the features are poorly represented, the network will not be able to generate them correctly. For instance, to increase the fidelity, there is the so-called <a href="https://paperswithcode.com/method/truncation-trick" rel="nofollow noreferrer">truncation trick</a>, where we sample the latent features vector <span class="math-container">$z$</span> from a truncated normal distribution, which ignores rare features. But on the other hand, it reduces diversity.</p>
  <p>As for the loss function,</p>
  <p><span class="math-container">$$\nabla_{\theta_{d}} \frac{1}{m} \sum_{i=1}^{m}\left[\log D\left(\boldsymbol{x}^{(i)}\right)+\log \left(1-D\left(G\left(\boldsymbol{z}^{(i)}\right)\right)\right)\right]$$</span></p>
  <p>the mean is associated with the loss rather than the iid of the random variables and is used to obtain the average gradient of a batch. Here are related questions: <a href="https://stats.stackexchange.com/questions/57628/why-is-the-logistic-regression-cost-function-scaled-by-the-number-of-examples">one</a>, <a href="https://math.stackexchange.com/questions/884887/why-divide-by-2m">tow</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-01T20:38:46.397Z
Answers.LastActivityDate: 2022-01-15T00:16:24.853Z
Answers.LastEditDate: 2022-01-15T00:16:24.853Z
Answers.UserDisplayName: Aray Karjauv
Body: <p>I am trying to understand <a href="https://ai.stackexchange.com/questions/23161/why-is-the-mean-used-to-compute-the-expectation-in-the-gan-loss">why mean is used for expectation</a> in training Generative Adversarial Networks.</p>
  <p>The answer tells that it is due to the <a href="https://www.probabilitycourse.com/chapter7/7_1_1_law_of_large_numbers.php" rel="nofollow noreferrer">law of large numbers</a> which is based on the assumption that random variables are independent and identically distributed.</p>
  <hr />
  <p>If I have a dataset of all possible <span class="math-container">$32 \times 32$</span> grayscale images. Then my sample space consists of <span class="math-container">$256^{32 \times 32}$</span> elements. Suppose I define 1024 random variables as</p>
  <p><span class="math-container">$$X_i = \text{ intensity of } i^{th} \text{ pixel for } 1 \le i \le 1024$$</span></p>
  <p>Then it is clear that all the random variables are iid since</p>
  <ol>
  <li><span class="math-container">$X_i \perp  X_j$</span> for all <span class="math-container">$i, j$</span> such that <span class="math-container">$i \ne j$</span> and</li>
  <li><span class="math-container">$p(X_i = k) = \dfrac{1}{256}$</span> for all <span class="math-container">$i$</span></li>
  </ol>
  <hr />
  <p>But these properties do not hold if I take a dataset of (say flower) images since pixel intensities are not independent of each other and the intensity values are not uniformly distributed as well.</p>
  <p>Then how can the law of large numbers be applicable for GAN as the dataset (sample space) does not cover all the possible elements? If I am wrong, then what is the sample space they are considering and what are the random variables they are using implicitly that leads to the satisfaction of iid condition and then the law of large numbers?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-01T14:52:45.523Z
Comments.Text: GANs do not learn pixels directly, but a mapping between latent features (e.g., color, rotation, width, etc.) and an output image.
Comments.UserDisplayName: Aray Karjauv
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-01T15:07:51.077Z
Comments.Text: @ArayKarjauv But the value function is saying $x∼p_{data}$ under the expectation, which then I understood $x$ as a random vector containing random variables which says the intensities of pixels. Is it wrong?
Comments.UserDisplayName: hanugm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-01T15:48:02.893Z
Comments.Text: $x$ is an image with abstract attributes, which a GAN aims to learn. Specifically, GAN tries to approximate a lower dimensional manifold that represents the real data. Of course, all modalities must be evenly represented in the dataset, but I think that the mean is rather related to the loss than to the iid of the random variables.
Comments.UserDisplayName: Aray Karjauv
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-01T18:54:34.690Z
Comments.Text: @ArayKarjauv Your second comment is close to a full answer IMO
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-01T21:39:40.593Z
Comments.Text: @ArayKarjauv So, is [this](https://ai.stackexchange.com/a/23163/18758) answer wrong?
Comments.UserDisplayName: hanugm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-01T22:26:42.687Z
Comments.Text: I find this answer incomplete.  It just discusses the expectation itself, as the author stated at the beginning.
Comments.UserDisplayName: Aray Karjauv
Tags: <math><generative-adversarial-networks><expectation><random-variable><iid>
Title: What are the iid random variables for a dataset in the GAN framework?
UserDisplayName: hanugm
ThreadText: # What are the iid random variables for a dataset in the GAN framework?
  
  (Asked by: username_0 on 2021-08-01)
  
  I am trying to understand [why mean is used for expectation](https://ai.stackexchange.com/questions/23161/why-is-the-mean-used-to-compute-the-expectation-in-the-gan-loss) in training Generative Adversarial Networks.
  
  The answer tells that it is due to the [law of large numbers](https://www.probabilitycourse.com/chapter7/7_1_1_law_of_large_numbers.php) which is based on the assumption that random variables are independent and identically distributed.
  
  ---
  
  If I have a dataset of all possible $32 \times 32$ grayscale images. Then my sample space consists of $256^{32 \times 32}$ elements. Suppose I define 1024 random variables as
  
  $$X\_i = \text{ intensity of } i^{th} \text{ pixel for } 1 \le i \le 1024$$
  
  Then it is clear that all the random variables are iid since
  
  1. $X\_i \perp X\_j$ for all $i, j$ such that $i \ne j$ and
  2. $p(X\_i = k) = \dfrac{1}{256}$ for all $i$
  
  ---
  
  But these properties do not hold if I take a dataset of (say flower) images since pixel intensities are not independent of each other and the intensity values are not uniformly distributed as well.
  
  Then how can the law of large numbers be applicable for GAN as the dataset (sample space) does not cover all the possible elements? If I am wrong, then what is the sample space they are considering and what are the random variables they are using implicitly that leads to the satisfaction of iid condition and then the law of large numbers?
  
  **Tags:** math, generative-adversarial-networks, expectation, random-variable, iid
  > **Comments:**
  > **username_1**: GANs do not learn pixels directly, but a mapping between latent features (e.g., color, rotation, width, etc.) and an output image.  
  > **username_0**: @ArayKarjauv But the value function is saying $x∼p_{data}$ under the expectation, which then I understood $x$ as a random vector containing random variables which says the intensities of pixels. Is it wrong?  
  > **username_1**: $x$ is an image with abstract attributes, which a GAN aims to learn. Specifically, GAN tries to approximate a lower dimensional manifold that represents the real data. Of course, all modalities must be evenly represented in the dataset, but I think that the mean is rather related to the loss than to the iid of the random variables.  
  > **username_2**: @ArayKarjauv Your second comment is close to a full answer IMO  
  > **username_0**: @ArayKarjauv So, is [this](https://ai.stackexchange.com/a/23163/18758) answer wrong?  
  > **username_1**: I find this answer incomplete.  It just discusses the expectation itself, as the author stated at the beginning.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-08-01. Score: 1
  
  [Independent and identically distributed](https://en.wikipedia.org/wiki/Independent_and_identically_distributed_random_variables) random variables share the same probability distribution and each item doesn’t influence or provide insight about the value of the next item you measure. The most common example is a coin toss: as you flip the coin, one outcome does not influence or predict the next one.
  
  As for a dataset of flowers, we assume that the photos were made independently and all kinds of flowers are evenly represented, which in practice is not true. Each flower won't be represented in the same lighting condition and camera view.
  
  Each pixel in an image is highly dependent on the neighboring pixels (and sometimes distant ones, e.g., when generating faces as they are usual symmetric) and the main goal of GAN is to discover this dependence by exploring a latent features manifold that represents abstract attributes of the real data, and to find a mapping between these features and output pixels. Of course, if some of the features are poorly represented, the network will not be able to generate them correctly. For instance, to increase the fidelity, there is the so-called [truncation trick](https://paperswithcode.com/method/truncation-trick), where we sample the latent features vector $z$ from a truncated normal distribution, which ignores rare features. But on the other hand, it reduces diversity.
  
  As for the loss function,
  
  $$\nabla\_{\theta\_{d}} \frac{1}{m} \sum\_{i=1}^{m}\left[\log D\left(\boldsymbol{x}^{(i)}\right)+\log \left(1-D\left(G\left(\boldsymbol{z}^{(i)}\right)\right)\right)\right]$$
  
  the mean is associated with the loss rather than the iid of the random variables and is used to obtain the average gradient of a batch. Here are related questions: [one](https://stats.stackexchange.com/questions/57628/why-is-the-logistic-regression-cost-function-scaled-by-the-number-of-examples), [tow](https://math.stackexchange.com/questions/884887/why-divide-by-2m).
  
  
  ---
  
  