Answers.Body: <p>Your goal is to model a distribution when constructing a GAN, therefore you need a way to be able to sample that distribution. The <em>noise</em>'s purpose is so you can do this. Generally, it's drawn from a distribution that is computationally easy to draw from (like a gaussian).   </p>
  
  <p>You are modeling the generator <span class="math-container">$G(X)$</span>  where <span class="math-container">$X \sim N(\mu, \sigma^2)$</span>. this means <span class="math-container">$G(X)$</span> is a random variable itself. The forward pass of the network transforms the <span class="math-container">$X$</span> samples into our <span class="math-container">$G(X)$</span> samples, allowing us to formulate a loss function (by solving the expectation as the mean of drawn samples) and train the model.  </p>
  
  <p><strong>Takeaway</strong>: The noise injected is just a parametrization of our Generator in another space, and the training goal is to learn the ideal transformation (we use neural networks because they are differentiable and are effective function approximators)  </p>
  
  <p>Also, note to your point of why it doesn't learn the training data (in its entirety) exactly is because generally <span class="math-container">$G(X)$</span> is continuous, and therefore if it has 2 images in its codomain, there also exists some path in pixel space from one to the other containing an uncountable (or quantized countable) number of images that don't exist in the training, and this would be reflected in the loss, therefore in the min-max game of the optimization, its difficult for it learn the training set on the nose.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-24T12:26:36.250Z
Answers.LastActivityDate: 2019-07-24T21:56:07.843Z
Answers.LastEditDate: 2019-07-24T21:56:07.843Z
Answers.UserDisplayName: mshlis
Body: <p>I do not understand why with enough training how the generator cannot learn all images from the training set as a mapping from the latent space - It is the absolute optimal case in training as it replicates the distribution and the discriminator output will always be 0.5. Even though most blog posts I have seen do not mention noise, a few of them have them in their diagrams or describe their presence, but never exactly describe the purpose of this noise. </p>
  
  <p>Is this noise injected to avoid the exact reproduction of the training data? If not what is the purpose of this injection and how is exact reproduction avoided?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-24T12:15:56.040Z
Comments.Text: where is noise "injected"? or do you mean how noise is used as an input
Comments.UserDisplayName: mshlis
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-24T12:18:54.127Z
Comments.Text: @mshlis of the generator networks I have come across the input vector was always the latent vector _**z**_ , whenever noise was injected it was usually added before the convolution operations
Comments.UserDisplayName: ashenoy
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-24T12:19:55.930Z
Comments.Text: generally **z** is a noise vector (sometime amended by some feature vecdtor)
Comments.UserDisplayName: mshlis
Tags: <deep-learning><generative-adversarial-networks><generative-model>
Title: What is the purpose of the noise injection in the generator network of a GAN?
UserDisplayName: ashenoy
ThreadText: # What is the purpose of the noise injection in the generator network of a GAN?
  
  (Asked by: username_0 on 2019-07-24)
  
  I do not understand why with enough training how the generator cannot learn all images from the training set as a mapping from the latent space - It is the absolute optimal case in training as it replicates the distribution and the discriminator output will always be 0.5. Even though most blog posts I have seen do not mention noise, a few of them have them in their diagrams or describe their presence, but never exactly describe the purpose of this noise.
  
  Is this noise injected to avoid the exact reproduction of the training data? If not what is the purpose of this injection and how is exact reproduction avoided?
  
  **Tags:** deep-learning, generative-adversarial-networks, generative-model
  
  > **Comments:**
  > **username_1**: where is noise "injected"? or do you mean how noise is used as an input  
  > **username_0**: @username_1 of the generator networks I have come across the input vector was always the latent vector _**z**_ , whenever noise was injected it was usually added before the convolution operations  
  > **username_1**: generally **z** is a noise vector (sometime amended by some feature vecdtor)  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-07-24. Score: 2 | Accepted answer
  
  Your goal is to model a distribution when constructing a GAN, therefore you need a way to be able to sample that distribution. The *noise*'s purpose is so you can do this. Generally, it's drawn from a distribution that is computationally easy to draw from (like a gaussian).
  
  You are modeling the generator $G(X)$ where $X \sim N(\mu, \sigma^2)$. this means $G(X)$ is a random variable itself. The forward pass of the network transforms the $X$ samples into our $G(X)$ samples, allowing us to formulate a loss function (by solving the expectation as the mean of drawn samples) and train the model.
  
  **Takeaway**: The noise injected is just a parametrization of our Generator in another space, and the training goal is to learn the ideal transformation (we use neural networks because they are differentiable and are effective function approximators)
  
  Also, note to your point of why it doesn't learn the training data (in its entirety) exactly is because generally $G(X)$ is continuous, and therefore if it has 2 images in its codomain, there also exists some path in pixel space from one to the other containing an uncountable (or quantized countable) number of images that don't exist in the training, and this would be reflected in the loss, therefore in the min-max game of the optimization, its difficult for it learn the training set on the nose.
  
  
  ---
  
  