Answers.Body: <p>In the source code, the author defines <code>sd</code> by</p>
  
  <pre><code>sd       = 0.5 * tf.layers.dense(x, units=n_latent)    
  </code></pre>
  
  <p>which means that <span class="math-container">$\operatorname{sd}\in \mathbb{R}^n$</span>.
  In particular, the support over <code>sd</code> includes negative numbers, which is something we want to avoid.
  Since standard deviations are always nonnegative, we can exponentiate to get us in the correct domain.
  This is a case where the variable is inappropriately named.
  Here, <code>sd</code> is not the standard deviation itself but rather the <em>logarithm</em> of the standard deviation. This allows it to be predicted as the output of a layer in a neural network, so extracting the predicted value of the standard deviation would require exponentiation.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-06T07:41:20.080Z
Answers.Text: hm yes, tks, that `sd` is `ln` of sd, not sd
Answers.UserDisplayName: Dan D.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-06T07:37:47.667Z
Answers.LastActivityDate: 2020-02-06T07:37:47.667Z
Answers.UserDisplayName: Anon
Body: <p>There's one VAE example here: 
  <a href="https://towardsdatascience.com/teaching-a-variational-autoencoder-vae-to-draw-mnist-characters-978675c95776" rel="nofollow noreferrer">https://towardsdatascience.com/teaching-a-variational-autoencoder-vae-to-draw-mnist-characters-978675c95776</a>.</p>
  
  <p>And the source code of encoder can be found at the following URL: <a href="https://gist.github.com/FelixMohr/29e1d5b1f3fd1b6374dfd3b68c2cdbac#file-vae-py" rel="nofollow noreferrer">https://gist.github.com/FelixMohr/29e1d5b1f3fd1b6374dfd3b68c2cdbac#file-vae-py</a>.</p>
  
  <p>The author is using <span class="math-container">$e$</span> (natural exponential) for calculating values of the embedding vector:</p>
  
  <p><span class="math-container">$$z = \mu + \epsilon \times e^{\sigma}$$</span></p>
  
  <p>where <span class="math-container">$\mu$</span> is the mean, <span class="math-container">$\epsilon$</span> a small random number and <span class="math-container">$\sigma$</span> the standard deviation.</p>
  
  <p>Or in code</p>
  
  <pre><code>z  = mn + tf.multiply(epsilon, tf.exp(sd))
  </code></pre>
  
  <p>It's not related to the code (practical programming), but why using natural exponential instead of:</p>
  
  <p><span class="math-container">$$z = \mu + \epsilon \times \sigma$$</span></p>
  
Tags: <machine-learning><autoencoders><variational-autoencoder><notation>
Title: Why is exp used in encoder of VAE instead of using the value of standard deviation alone?
UserDisplayName: Dan D.
ThreadText: # Why is exp used in encoder of VAE instead of using the value of standard deviation alone?
  
  (Asked by: username_0 on 2020-02-06)
  
  There's one VAE example here:
  <https://towardsdatascience.com/teaching-a-variational-autoencoder-vae-to-draw-mnist-characters-978675c95776>.
  
  And the source code of encoder can be found at the following URL: <https://gist.github.com/FelixMohr/29e1d5b1f3fd1b6374dfd3b68c2cdbac#file-vae-py>.
  
  The author is using $e$ (natural exponential) for calculating values of the embedding vector:
  
  $$z = \mu + \epsilon \times e^{\sigma}$$
  
  where $\mu$ is the mean, $\epsilon$ a small random number and $\sigma$ the standard deviation.
  
  Or in code
  
  ```
  z  = mn + tf.multiply(epsilon, tf.exp(sd))
  
  ```
  
  It's not related to the code (practical programming), but why using natural exponential instead of:
  
  $$z = \mu + \epsilon \times \sigma$$
  
  **Tags:** machine-learning, autoencoders, variational-autoencoder, notation
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-02-06. Score: 2 | Accepted answer
  
  In the source code, the author defines `sd` by
  
  ```
  sd       = 0.5 * tf.layers.dense(x, units=n_latent)    
  
  ```
  
  which means that $\operatorname{sd}\in \mathbb{R}^n$.
  In particular, the support over `sd` includes negative numbers, which is something we want to avoid.
  Since standard deviations are always nonnegative, we can exponentiate to get us in the correct domain.
  This is a case where the variable is inappropriately named.
  Here, `sd` is not the standard deviation itself but rather the *logarithm* of the standard deviation. This allows it to be predicted as the output of a layer in a neural network, so extracting the predicted value of the standard deviation would require exponentiation.
  
  > **Comments:**
  > **username_0**: hm yes, tks, that `sd` is `ln` of sd, not sd  
  
  ---
  
  