Answers.Body: <p>In the source code, the author defines <code>sd</code> by</p>
  
  <pre><code>sd       = 0.5 * tf.layers.dense(x, units=n_latent)    
  </code></pre>
  
  <p>which means that <span class="math-container">$\operatorname{sd}\in \mathbb{R}^n$</span>.
  In particular, the support over <code>sd</code> includes negative numbers, which is something we want to avoid.
  Since standard deviations are always nonnegative, we can exponentiate to get us in the correct domain.
  This is a case where the variable is inappropriately named.
  Here, <code>sd</code> is not the standard deviation itself but rather the <em>logarithm</em> of the standard deviation. This allows it to be predicted as the output of a layer in a neural network, so extracting the predicted value of the standard deviation would require exponentiation.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-06T07:41:20.080Z
Answers.Text: hm yes, tks, that `sd` is `ln` of sd, not sd
Answers.UserDisplayName: Dan D.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-06T07:37:47.667Z
Answers.LastActivityDate: 2020-02-06T07:37:47.667Z
Answers.UserDisplayName: Anon
Body: <p>There's one VAE example here: 
  <a href="https://towardsdatascience.com/teaching-a-variational-autoencoder-vae-to-draw-mnist-characters-978675c95776" rel="nofollow noreferrer">https://towardsdatascience.com/teaching-a-variational-autoencoder-vae-to-draw-mnist-characters-978675c95776</a>.</p>
  
  <p>And the source code of encoder can be found at the following URL: <a href="https://gist.github.com/FelixMohr/29e1d5b1f3fd1b6374dfd3b68c2cdbac#file-vae-py" rel="nofollow noreferrer">https://gist.github.com/FelixMohr/29e1d5b1f3fd1b6374dfd3b68c2cdbac#file-vae-py</a>.</p>
  
  <p>The author is using <span class="math-container">$e$</span> (natural exponential) for calculating values of the embedding vector:</p>
  
  <p><span class="math-container">$$z = \mu + \epsilon \times e^{\sigma}$$</span></p>
  
  <p>where <span class="math-container">$\mu$</span> is the mean, <span class="math-container">$\epsilon$</span> a small random number and <span class="math-container">$\sigma$</span> the standard deviation.</p>
  
  <p>Or in code</p>
  
  <pre><code>z  = mn + tf.multiply(epsilon, tf.exp(sd))
  </code></pre>
  
  <p>It's not related to the code (practical programming), but why using natural exponential instead of:</p>
  
  <p><span class="math-container">$$z = \mu + \epsilon \times \sigma$$</span></p>
  
Tags: <machine-learning><autoencoders><variational-autoencoder><notation>
Title: Why is exp used in encoder of VAE instead of using the value of standard deviation alone?
UserDisplayName: Dan D.
ThreadText: # Why is exp used in encoder of VAE instead of using the value of standard deviation alone?
  
  (Asked by: username_0 on 2020-02-06)
  
  There's one VAE example here: 
  https://towardsdatascience.com/teaching-a-variational-autoencoder-vae-to-draw-mnist-characters-978675c95776.
  And the source code of encoder can be found at the following URL: https://gist.github.com/FelixMohr/29e1d5b1f3fd1b6374dfd3b68c2cdbac#file-vae-py.
  The author is using $e$ (natural exponential) for calculating values of the embedding vector:
  $$z = \mu + \epsilon \times e^{\sigma}$$
  where $\mu$ is the mean, $\epsilon$ a small random number and $\sigma$ the standard deviation.
  Or in code
  z  = mn + tf.multiply(epsilon, tf.exp(sd))
  
  It's not related to the code (practical programming), but why using natural exponential instead of:
  $$z = \mu + \epsilon \times \sigma$$
  
  **Tags:** machine-learning, autoencoders, variational-autoencoder, notation
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-02-06. Score: 2 | Accepted answer
  
  In the source code, the author defines sd by
  sd       = 0.5 * tf.layers.dense(x, units=n_latent)    
  
  which means that $\operatorname{sd}\in \mathbb{R}^n$.
  In particular, the support over sd includes negative numbers, which is something we want to avoid.
  Since standard deviations are always nonnegative, we can exponentiate to get us in the correct domain.
  This is a case where the variable is inappropriately named.
  Here, sd is not the standard deviation itself but rather the logarithm of the standard deviation. This allows it to be predicted as the output of a layer in a neural network, so extracting the predicted value of the standard deviation would require exponentiation.
  
  > **Comments:**
  > **username_0**: hm yes, tks, that `sd` is `ln` of sd, not sd  
  
  ---
  
  