Answers.Body: <p>It seems your question is concerned with how an empirical mean works.</p>
  <p>It is indeed true that, if all <span class="math-container">$x^{(i)}$</span> are independent identically distributed realisations of a random variable <span class="math-container">$X$</span>, then <span class="math-container">$\lim_{n \rightarrow \infty} \frac{1}{n}\sum_{i=1}^n f(x^{(i)}) = \mathbb{E}[f(X)]$</span>. This is a standard result in statistics known as <a href="https://en.wikipedia.org/wiki/Law_of_large_numbers#History" rel="nofollow noreferrer">the law of large numbers</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-21T10:15:56.273Z
Answers.LastActivityDate: 2020-08-21T10:15:56.273Z
Answers.UserDisplayName: David
Body: <p>From <a href="https://arxiv.org/abs/1406.2661" rel="nofollow noreferrer">Goodfellow et al. (2014)</a>, we have the adversarial loss:</p>
  <p><span class="math-container">$$ \min_G \, \max_D V (D, G) = \mathbb{E}_{x∼p_{data}(x)} \, [\log \, D(x)]  + \, \mathbb{E}_{z∼p_z(z)} \, [\log \, (1 − D(G(z)))] \, \text{.} \quad$$</span></p>
  <p>In practice, the expectation is computed as a mean over the minibatch. For example, the discriminator loss is:</p>
  <p><span class="math-container">$$
  \nabla_{\theta_{d}} \frac{1}{m} \sum_{i=1}^{m}\left[\log D\left(\boldsymbol{x}^{(i)}\right)+\log \left(1-D\left(G\left(\boldsymbol{z}^{(i)}\right)\right)\right)\right]
  $$</span></p>
  <p>My question is: <strong>why is the mean used to compute the expectation</strong>? Does this imply that <span class="math-container">$p_{data}$</span> is <strong>uniformly distributed</strong>, since every sample must be drawn from <span class="math-container">$p_{data}$</span> with equal probability?</p>
  <p>The expectation, expressed as an integral, is:</p>
  <p><span class="math-container">$$
  \begin{aligned}
  V(G, D) &amp;=\int_{\boldsymbol{x}} p_{\text {data }}(\boldsymbol{x}) \log (D(\boldsymbol{x})) d x+\int_{\boldsymbol{z}} p_{\boldsymbol{z}}(\boldsymbol{z}) \log (1-D(g(\boldsymbol{z}))) d z \\
  &amp;=\int_{\boldsymbol{x}} p_{\text {data }}(\boldsymbol{x}) \log (D(\boldsymbol{x}))+p_{g}(\boldsymbol{x}) \log (1-D(\boldsymbol{x})) d x
  \end{aligned}
  $$</span></p>
  <p><strong>So, how do we go from an integral involving a continuous distribution to summing over discrete probabilities, and further, that all those probabilities are the same?</strong></p>
  <p>The best I could find from other StackExchange posts is that the mean is just an approximation, but I'd really like a more rigorous explanation.</p>
  <p>This question isn't exclusive to GANs, but is applicable to any loss function that is expressed mathematically as an expectation over some sampled distribution, which is not implemented directly via the integral form.</p>
  <p>(All equations are from the Goodfellow paper.)</p>
  
Tags: <deep-learning><objective-functions><generative-adversarial-networks><expectation>
Title: Why is the mean used to compute the expectation in the GAN loss?
UserDisplayName: A is for Ambition
ThreadText: # Why is the mean used to compute the expectation in the GAN loss?
  
  (Asked by: username_0 on 2020-08-21)
  
  From [Goodfellow et al. (2014)](https://arxiv.org/abs/1406.2661), we have the adversarial loss:
  
  $$ \min\_G \, \max\_D V (D, G) = \mathbb{E}\_{x∼p\_{data}(x)} \, [\log \, D(x)] + \, \mathbb{E}\_{z∼p\_z(z)} \, [\log \, (1 − D(G(z)))] \, \text{.} \quad$$
  
  In practice, the expectation is computed as a mean over the minibatch. For example, the discriminator loss is:
  
  $$
  \nabla\_{\theta\_{d}} \frac{1}{m} \sum\_{i=1}^{m}\left[\log D\left(\boldsymbol{x}^{(i)}\right)+\log \left(1-D\left(G\left(\boldsymbol{z}^{(i)}\right)\right)\right)\right]
  $$
  
  My question is: **why is the mean used to compute the expectation**? Does this imply that $p\_{data}$ is **uniformly distributed**, since every sample must be drawn from $p\_{data}$ with equal probability?
  
  The expectation, expressed as an integral, is:
  
  $$
  \begin{aligned}
  V(G, D) &=\int\_{\boldsymbol{x}} p\_{\text {data }}(\boldsymbol{x}) \log (D(\boldsymbol{x})) d x+\int\_{\boldsymbol{z}} p\_{\boldsymbol{z}}(\boldsymbol{z}) \log (1-D(g(\boldsymbol{z}))) d z \\
  &=\int\_{\boldsymbol{x}} p\_{\text {data }}(\boldsymbol{x}) \log (D(\boldsymbol{x}))+p\_{g}(\boldsymbol{x}) \log (1-D(\boldsymbol{x})) d x
  \end{aligned}
  $$
  
  **So, how do we go from an integral involving a continuous distribution to summing over discrete probabilities, and further, that all those probabilities are the same?**
  
  The best I could find from other StackExchange posts is that the mean is just an approximation, but I'd really like a more rigorous explanation.
  
  This question isn't exclusive to GANs, but is applicable to any loss function that is expressed mathematically as an expectation over some sampled distribution, which is not implemented directly via the integral form.
  
  (All equations are from the Goodfellow paper.)
  
  **Tags:** deep-learning, objective-functions, generative-adversarial-networks, expectation
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-08-21. Score: 4 | Accepted answer
  
  It seems your question is concerned with how an empirical mean works.
  
  It is indeed true that, if all $x^{(i)}$ are independent identically distributed realisations of a random variable $X$, then $\lim\_{n \rightarrow \infty} \frac{1}{n}\sum\_{i=1}^n f(x^{(i)}) = \mathbb{E}[f(X)]$. This is a standard result in statistics known as [the law of large numbers](https://en.wikipedia.org/wiki/Law_of_large_numbers#History).
  
  
  ---
  
  