Answers.Body: <p>Mean <span class="math-container">$\mu(x)$</span> and standard deviation <span class="math-container">$\sigma(x)$</span> are actually learnable functions, whose parameters are adjusted via the back propagation procedure.</p>
  <p>Mean and standard deviation are not computed on the input vector <span class="math-container">$x$</span> or any transform of it.</p>
  <p>The procedure is the following:</p>
  <ul>
  <li>Pass the sample <span class="math-container">$x$</span> from the training data</li>
  </ul>
  <ul>
  <li>Propagate this vector <span class="math-container">$x$</span> through some NN (Feedforward MLP) and obtain some other vector <span class="math-container">$\tilde{x}$</span></li>
  <li>Get the mean <span class="math-container">$\mu(x)$</span> and std <span class="math-container">$\sigma(x)$</span> from <span class="math-container">$\tilde{x}$</span> from two more neural networks (maybe single layer)</li>
  </ul>
  <ul>
  <li>Generate random noise <span class="math-container">$\varepsilon$</span> and get a point in the latent space <span class="math-container">$\mu(x) + \sigma(x) \varepsilon$</span> (it is known as reparametrization trick)</li>
  </ul>
  <p><a href="https://i.sstatic.net/wloBb.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/wloBb.png" alt="enter image description here" /></a></p>
  <p>You can think about the procedure as follows - you have Normal distribution around each point of the input data in the latent space, and the mean <span class="math-container">$\mu(x)$</span> and <span class="math-container">$\sigma(x)$</span> are the parameters of this distribution (different for each point). The generated data is expected to resemble the training example, but differ in some reasonable sense, belong to the manifold of realistic images.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-17T01:42:31.340Z
Answers.Text: But, what is the gaurentee for us that those single feed forward layers are indeed calculating mean and standard deviation? Is there any reason behind calling them as **mean** and **standard deviation**?
Answers.UserDisplayName: hanugm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-17T06:05:40.547Z
Answers.Text: @hanugm they are defined to do this. Given any reference sample $x$ during many enough forward passes with the fixed parameters of networks $\mu(x)$ and $\sigma(x)$ - the point cloud to which this point will be mapped will have mean $\mu(x)$ and standard deviation $\sigma(x)$. It is guaranteed by the reparametrization trick.
Answers.UserDisplayName: spiridon_the_sun_rotator
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-16T06:24:32.627Z
Answers.LastActivityDate: 2021-08-16T06:24:32.627Z
Answers.UserDisplayName: spiridon_the_sun_rotator
Answers.Body: <p>The answer relies in the KL item of the loss function -
  it regularizes (=enforce) the distribution of z (represented by mu and sigma)
  to be close to N(0,1).</p>
  <p>That's the meaning of the KL divergence - a distance measure between probabilities and in order to minimize that penalty item - the network will tune it's parameters such that the z would be very close to N(0,1).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-18T10:44:05.000Z
Answers.LastActivityDate: 2022-03-18T10:44:05.000Z
Answers.UserDisplayName: eli
Body: <p>In general, if I have a collection of data then mean(Expectation) and standard deviation are calculated as follows</p>
  <p><span class="math-container">$$\text{mean } = \mu = \mathbb{E}[X] = \sum\limits_{i = 1}^n p_ix_i $$</span>
  <span class="math-container">$$\text{Variance =}\sigma (X) = \sqrt{\sum\limits_{i = 1}^{n}p_i{(x_i - \mu)^2}{}}$$</span></p>
  <p>where <span class="math-container">$X$</span> is a random vector having support <span class="math-container">$\{x_1, x_2, x_3, \cdots, x_n\}$</span>.</p>
  <p>Thus a dataset of samples have a single mean and single variance.</p>
  <p>Now, let us discuss about the case of variational auto-encoders. They look like follows
  <a href="https://i.sstatic.net/vlghb.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/vlghb.png" alt="enter image description here" /></a></p>
  <p>Suppose I trained the above auto-encoder on a training set, then for each sample I will get a mean and standard deviation at latent layer. Here, we can get a new <span class="math-container">$\mu$</span> and <span class="math-container">$\sigma$</span>  for each data sample. But, as we see earlier, mean and standard deviation exists for a dataset and not for each sample.</p>
  <p>I am confused about &quot;how can we say that mean and standard deviation are obtained at latent layer if they are not constant in nature&quot;?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-18T11:05:00.177Z
Comments.Text: I don't want to edit your question, but I believe you have put "Variance" where you should have put "Standard Deviation" in the formula with $\sigma (X)$.  However, the formula on the right hand side isn't quite right for stddev (or certainly for variance)
Comments.UserDisplayName: David Hoelzer
Tags: <autoencoders><variational-autoencoder>
Title: Are mean and standard deviation in variational autoencoders unique?
UserDisplayName: hanugm
ThreadText: # Are mean and standard deviation in variational autoencoders unique?
  
  (Asked by: username_0 on 2021-08-16)
  
  In general, if I have a collection of data then mean(Expectation) and standard deviation are calculated as follows
  
  $$\text{mean } = \mu = \mathbb{E}[X] = \sum\limits\_{i = 1}^n p\_ix\_i $$
  $$\text{Variance =}\sigma (X) = \sqrt{\sum\limits\_{i = 1}^{n}p\_i{(x\_i - \mu)^2}{}}$$
  
  where $X$ is a random vector having support $\{x\_1, x\_2, x\_3, \cdots, x\_n\}$.
  
  Thus a dataset of samples have a single mean and single variance.
  
  Now, let us discuss about the case of variational auto-encoders. They look like follows
  [![enter image description here](https://i.sstatic.net/vlghb.png)](https://i.sstatic.net/vlghb.png)
  
  Suppose I trained the above auto-encoder on a training set, then for each sample I will get a mean and standard deviation at latent layer. Here, we can get a new $\mu$ and $\sigma$ for each data sample. But, as we see earlier, mean and standard deviation exists for a dataset and not for each sample.
  
  I am confused about "how can we say that mean and standard deviation are obtained at latent layer if they are not constant in nature"?
  
  **Tags:** autoencoders, variational-autoencoder
  
  > **Comments:**
  > **username_1**: I don't want to edit your question, but I believe you have put "Variance" where you should have put "Standard Deviation" in the formula with $\sigma (X)$.  However, the formula on the right hand side isn't quite right for stddev (or certainly for variance)  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2021-08-16. Score: 1
  
  Mean $\mu(x)$ and standard deviation $\sigma(x)$ are actually learnable functions, whose parameters are adjusted via the back propagation procedure.
  
  Mean and standard deviation are not computed on the input vector $x$ or any transform of it.
  
  The procedure is the following:
  
  * Pass the sample $x$ from the training data
  
  * Propagate this vector $x$ through some NN (Feedforward MLP) and obtain some other vector $\tilde{x}$
  * Get the mean $\mu(x)$ and std $\sigma(x)$ from $\tilde{x}$ from two more neural networks (maybe single layer)
  
  * Generate random noise $\varepsilon$ and get a point in the latent space $\mu(x) + \sigma(x) \varepsilon$ (it is known as reparametrization trick)
  
  [![enter image description here](https://i.sstatic.net/wloBb.png)](https://i.sstatic.net/wloBb.png)
  
  You can think about the procedure as follows - you have Normal distribution around each point of the input data in the latent space, and the mean $\mu(x)$ and $\sigma(x)$ are the parameters of this distribution (different for each point). The generated data is expected to resemble the training example, but differ in some reasonable sense, belong to the manifold of realistic images.
  
  > **Comments:**
  > **username_0**: But, what is the gaurentee for us that those single feed forward layers are indeed calculating mean and standard deviation? Is there any reason behind calling them as **mean** and **standard deviation**?  
  > **username_2**: @username_0 they are defined to do this. Given any reference sample $x$ during many enough forward passes with the fixed parameters of networks $\mu(x)$ and $\sigma(x)$ - the point cloud to which this point will be mapped will have mean $\mu(x)$ and standard deviation $\sigma(x)$. It is guaranteed by the reparametrization trick.  
  
  ---
  
  ### Answer by username_3 on 2022-03-18. Score: 0
  
  The answer relies in the KL item of the loss function -
  it regularizes (=enforce) the distribution of z (represented by mu and sigma)
  to be close to N(0,1).
  
  That's the meaning of the KL divergence - a distance measure between probabilities and in order to minimize that penalty item - the network will tune it's parameters such that the z would be very close to N(0,1).
  
  
  ---
  
  