Answers.Body: <p>As pointed out in the comments, a random variable <span class="math-container">$z$</span> sampled from a uniform distribution <span class="math-container">$\mathcal{U}(a,b)$</span>, doesn't take only two values but <strong>any</strong> value between <span class="math-container">$a$</span> and <span class="math-container">$b$</span> with equal probability <span class="math-container">$\frac{1}{b-a}$</span>.</p>
  <p>The distribution you use for <span class="math-container">$z$</span> in a GAN doesn't have a theoretical justification. The fact that it's an <span class="math-container">$\mathcal{N}(0,1)$</span> is never used in the proof of the convergence of the generator to the data distribution in the <a href="https://arxiv.org/pdf/1406.2661.pdf" rel="nofollow noreferrer">original GAN paper</a> (see Section 4.1).</p>
  <p>However, it should be easy and fast to sample from given that you must take samples at each training step. Why certain authors choose one distribution over the other is usually down to empirical trial and error. GANs are notoriously hard to train and the distribution of <span class="math-container">$z$</span> is one of the many hyperparameters of the model.</p>
  <p>Other generative models though have theoretical justification for the choice of the distribution of the random variable given to the decoder. In VAEs, it is a Gaussian <span class="math-container">$\mathcal{N}(0,1)$</span> because the loss function contains a KL term <span class="math-container">$\text{KL}(p_\theta(z|x)||p(z))$</span> which has a nice closed form when <span class="math-container">$p_\theta(z|x)$</span> and <span class="math-container">$p(z)$</span> are Gaussian distributions (see Section 3 in the <a href="https://arxiv.org/pdf/1312.6114.pdf" rel="nofollow noreferrer">original VAE paper</a>).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-18T20:55:23.737Z
Answers.LastActivityDate: 2022-05-18T20:55:23.737Z
Answers.UserDisplayName: Raphael Lopez Kaufman
Answers.Body: <p>The main idea of the latent vector <span class="math-container">$z$</span> is that it represents some high-level features of images, and the interpretation could be that latent features are normally distributed.</p>
  <p>However, since we are trying to approximate (map) a complex multimodal real distribution using a simple one, the feature space <span class="math-container">$Z$</span> becomes entangled. So, in practice, we just feed random noise into the generator to add stochasticity. Otherwise, the network would produce the same output each time.</p>
  <p>As already discussed in the comments, it doesn't really matter which distribution you use. The only advantage I see of using a normal distribution is that we can use the truncation trick to get better (but less varied) results.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-18T22:35:49.977Z
Answers.LastActivityDate: 2022-05-18T22:35:49.977Z
Answers.UserDisplayName: Aray Karjauv
Body: <p>In a Generative Adversarial Network (GAN), there are two multi-layer perceptrons. One is the generator network and another is a discriminator network.</p>
  <p>The input for the generator network is a noise vector <span class="math-container">$z$</span>. The input for a discriminator network is either a generated sample <span class="math-container">$G(z)$</span> i.e., the output of a generator network or a training sample <span class="math-container">$x$</span> for a training dataset.</p>
  <p>My doubt is regarding the input of the generator. The noise vector is generally sampled from the standard normal distribution.</p>
  <p><span class="math-container">$$z \sim \mathcal{N(0, 1)}$$</span></p>
  <p>Although I am not sure, I think : <em>since the values in the normal distribution vary, the output of the generator can vary accordingly.</em></p>
  <p>But some of the research papers say that the noise vector can also be sampled from a uniform distribution i.e., <span class="math-container">$z \sim \mathcal{U(a, b)}$</span> for <span class="math-container">$a&lt;b$</span>.</p>
  <p><span class="math-container">$$ U(x) = \begin{cases} 
        \dfrac{1}{b-a} &amp; x\in [a, b] \\
        0 &amp; x\not\in [a, b] \\
     \end{cases}
  $$</span></p>
  <p>It is clear that uniform distribution does not vary like normal distribution and takes only two possible values, hence all samples have equal probability in the given range. Then how can it contribute to the diversity of the output of the generator network?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-12T08:14:23.663Z
Comments.Text: Samples from the uniform distribution do not take only two possible values. And why do you think  "diversity" requires a non-uniform distribution?
Comments.UserDisplayName: Criticize SE actions means ban
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-12T08:15:10.490Z
Comments.Text: "The probability of rolling a 1 is 1/6. The probability of rolling a 2 is 1/6, and so on. So does that mean the only number I can get from a dice is 1/6?"
Comments.UserDisplayName: Criticize SE actions means ban
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-12T08:54:16.797Z
Comments.Text: **hence all samples have equal probability in the given range** but I am confused what is the need of such constant distribution @user253751
Comments.UserDisplayName: hanugm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-12T08:54:37.913Z
Comments.Text: What is the need of a normal distribution @hanugm?
Comments.UserDisplayName: Criticize SE actions means ban
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-12T08:56:34.450Z
Comments.Text: @user253751 because it provides samples with varying probability values.
Comments.UserDisplayName: hanugm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-12T09:02:25.660Z
Comments.Text: What is the need for samples with varying probability values? It is easy to construct varying probability values from fixed probability values using a nonlinear transformation.
Comments.UserDisplayName: Criticize SE actions means ban
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-12T10:02:23.190Z
Comments.Text: @user253751 Then according to you, any random probability distribution will work. Am I right?
Comments.UserDisplayName: hanugm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-12T10:52:35.363Z
Comments.Text: Probably.......
Comments.UserDisplayName: Criticize SE actions means ban
Tags: <generator><normal-distribution><uniform-distribution><noise>
Title: How does noise samples from uniform distribution contribute to the diversity of generator output?
UserDisplayName: hanugm
ThreadText: # How does noise samples from uniform distribution contribute to the diversity of generator output?
  
  (Asked by: username_0 on 2021-08-11)
  
  In a Generative Adversarial Network (GAN), there are two multi-layer perceptrons. One is the generator network and another is a discriminator network.
  
  The input for the generator network is a noise vector $z$. The input for a discriminator network is either a generated sample $G(z)$ i.e., the output of a generator network or a training sample $x$ for a training dataset.
  
  My doubt is regarding the input of the generator. The noise vector is generally sampled from the standard normal distribution.
  
  $$z \sim \mathcal{N(0, 1)}$$
  
  Although I am not sure, I think : *since the values in the normal distribution vary, the output of the generator can vary accordingly.*
  
  But some of the research papers say that the noise vector can also be sampled from a uniform distribution i.e., $z \sim \mathcal{U(a, b)}$ for $a<b$.
  
  $$ U(x) = \begin{cases}
  \dfrac{1}{b-a} & x\in [a, b] \\
  0 & x\not\in [a, b] \\
  \end{cases}
  $$
  
  It is clear that uniform distribution does not vary like normal distribution and takes only two possible values, hence all samples have equal probability in the given range. Then how can it contribute to the diversity of the output of the generator network?
  
  **Tags:** generator, normal-distribution, uniform-distribution, noise
  
  > **Comments:**
  > **username_1**: Samples from the uniform distribution do not take only two possible values. And why do you think  "diversity" requires a non-uniform distribution?  
  > **username_1**: "The probability of rolling a 1 is 1/6. The probability of rolling a 2 is 1/6, and so on. So does that mean the only number I can get from a dice is 1/6?"  
  > **username_0**: **hence all samples have equal probability in the given range** but I am confused what is the need of such constant distribution @user253751  
  > **username_1**: What is the need of a normal distribution @username_0?  
  > **username_0**: @user253751 because it provides samples with varying probability values.  
  > **username_1**: What is the need for samples with varying probability values? It is easy to construct varying probability values from fixed probability values using a nonlinear transformation.  
  > **username_0**: @user253751 Then according to you, any random probability distribution will work. Am I right?  
  > **username_1**: Probably.......  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2022-05-18. Score: 1
  
  As pointed out in the comments, a random variable $z$ sampled from a uniform distribution $\mathcal{U}(a,b)$, doesn't take only two values but **any** value between $a$ and $b$ with equal probability $\frac{1}{b-a}$.
  
  The distribution you use for $z$ in a GAN doesn't have a theoretical justification. The fact that it's an $\mathcal{N}(0,1)$ is never used in the proof of the convergence of the generator to the data distribution in the [original GAN paper](https://arxiv.org/pdf/1406.2661.pdf) (see Section 4.1).
  
  However, it should be easy and fast to sample from given that you must take samples at each training step. Why certain authors choose one distribution over the other is usually down to empirical trial and error. GANs are notoriously hard to train and the distribution of $z$ is one of the many hyperparameters of the model.
  
  Other generative models though have theoretical justification for the choice of the distribution of the random variable given to the decoder. In VAEs, it is a Gaussian $\mathcal{N}(0,1)$ because the loss function contains a KL term $\text{KL}(p\_\theta(z|x)||p(z))$ which has a nice closed form when $p\_\theta(z|x)$ and $p(z)$ are Gaussian distributions (see Section 3 in the [original VAE paper](https://arxiv.org/pdf/1312.6114.pdf)).
  
  
  ---
  
  ### Answer by username_3 on 2022-05-18. Score: 0
  
  The main idea of the latent vector $z$ is that it represents some high-level features of images, and the interpretation could be that latent features are normally distributed.
  
  However, since we are trying to approximate (map) a complex multimodal real distribution using a simple one, the feature space $Z$ becomes entangled. So, in practice, we just feed random noise into the generator to add stochasticity. Otherwise, the network would produce the same output each time.
  
  As already discussed in the comments, it doesn't really matter which distribution you use. The only advantage I see of using a normal distribution is that we can use the truncation trick to get better (but less varied) results.
  
  
  ---
  
  