Answers.Body: <p>Let's assume the probability distributions are Gaussian (or normal) distributions. In other words, in the Bayes' rule</p>
  
  <p><span class="math-container">\begin{align}
  p(z|x)=\frac{p(x|z)p(z)}{p(x)}
  \tag{1}\label{1}
  \end{align}</span></p>
  
  <p>The posterior <span class="math-container">$p(z|x)$</span>, the likelihood <span class="math-container">$p(x|z)$</span>, the prior <span class="math-container">$p(z)$</span> and the evidence (or marginal) <span class="math-container">$p(x)$</span> are Gaussian distributions. You can assume this because <a href="https://distill.pub/2019/visual-exploration-gaussian-processes/#Multivariate" rel="nofollow noreferrer">Gaussian distributions are closed under conditioning and marginalization</a>.</p>
  
  <p>For simplicity, let's further assume that they are <a href="https://en.wikipedia.org/wiki/Normal_distribution" rel="nofollow noreferrer"><strong>univariate</strong> Gaussian distributions</a>. Given that the Gaussian distribution is a <em>continuous probability distribution</em>, it has an associated <em>probability density function</em> (rather than a probability mass function, which is associated with discrete probability distributions, such as the <a href="https://en.wikipedia.org/wiki/Bernoulli_distribution" rel="nofollow noreferrer">Bernoulli distribution</a>). The probability density function of the Gaussian distribution is</p>
  
  <p><span class="math-container">\begin{align}
  f(x \mid \mu, \sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2} } e^{ -\frac{(x-\mu)^2}{2\sigma^2} } \tag{2}\label{2}
  \end{align}</span></p>
  
  <p>where <span class="math-container">$\mu$</span> and <span class="math-container">$\sigma^2$</span> are respectively the mean and variance of the Gaussian distribution and <span class="math-container">$x$</span> is a variable (similarly to a variable <span class="math-container">$x$</span> in any mathematical function <span class="math-container">$f(x)$</span>). So, given a concrete value for <span class="math-container">$x$</span>, for example, <span class="math-container">$x=1$</span>, then <span class="math-container">$f(x=1, \mu, \sigma^2)$</span> is a so-called <strong>density</strong> value (rather than a probability, which a probability mass function returns, given an input). For example, let's assume that the mean <span class="math-container">$\mu=0$</span> and the variance <span class="math-container">$\sigma^2 = 1$</span>, then, for <span class="math-container">$x=1$</span>, the density will be</p>
  
  <p><span class="math-container">$$
  f(1 \mid 0, 1) = \frac{1}{\sqrt{2\pi} } e^{ -\frac{1}{2} }
  $$</span></p>
  
  <p>So, to obtain the concrete density value, I've just replaced the concrete values of <span class="math-container">$x$</span>, <span class="math-container">$\mu$</span> and <span class="math-container">$\sigma^2$</span> in equation \ref{2}. </p>
  
  <p>To calculate the posterior <span class="math-container">$p(z|x)$</span> in equation \ref{1}, you just need to replace the likelihood <span class="math-container">$p(x|z)$</span>, the prior <span class="math-container">$p(z)$</span> and the evidence <span class="math-container">$p(x)$</span> with the Gaussian probability density shown in equation \ref{2}, so you will have</p>
  
  <p><span class="math-container">\begin{align}
  p(z|x)=\frac{f_{X\mid Z}(x \mid \mu_{X\mid Z}, \sigma^2_{X\mid Z}, z) f_{Z}(z \mid \mu_{Z}, \sigma^2_{Z})}{f_{X}(x \mid \mu_{X}, \sigma^2_{X})}
  \tag{3}\label{3}
  \end{align}</span></p>
  
  <p>I've explicitly added a subscript to the means and variances of each probability density, given that, for example, the mean of the probability density <span class="math-container">$f_{X\mid Z}$</span> might be different than the mean of the probability density <span class="math-container">$f_{Z}$</span> or <span class="math-container">$f_{X}$</span>, etc. So, to get the actual density value (a real number) that represents <span class="math-container">$p(z|x)$</span>, you just need to replace <span class="math-container">$f_{X\mid Z}$</span>, <span class="math-container">$f_{Z}$</span> and <span class="math-container">$f_{X}$</span> with the definition of the Gaussian density function in \ref{2} with their actual mean and variance values. I'll let you do this, given that this is really just a matter of picking up a concrete value for the means and variances and doing some algebra.</p>
  
  <p>If you assume the posterior, the likelihood, the prior or the evidence to have a different distribution, you will do the same thing, but using the probability density or mass function of your chosen distribution.</p>
  
  <p>In the context of the variational auto-encoder, you will be learning the mean and variance of the distribution, so the mean and variance will not be fixed, but they will be the parameters that you want to find. However, this does not change the way you apply the Bayes' rule.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-22T16:50:58.203Z
Answers.LastActivityDate: 2019-11-22T19:24:34.210Z
Answers.LastEditDate: 2019-11-22T19:24:34.210Z
Answers.UserDisplayName: nbro
Body: <p>In the context of the variational auto-encoder, can someone give me a concrete example of the application of the Bayes' rule</p>
  
  <p><span class="math-container">$$p_{\theta}(z|x)=\frac{p_{\theta}(x|z)p(z)}{p(x)}$$</span></p>
  
  <p>for a given latent variable and observable?</p>
  
  <p>I understand with VAE's we're essentially getting an approximation to <span class="math-container">$p_{\theta}(z|x)$</span> that models the distribution that we think approximates the latent variables, but I need a concrete example to really understand it.</p>
  
Tags: <convolutional-neural-networks><autoencoders><variational-autoencoder>
Title: Concrete example of latent variables and observables plugged into the Bayes' rule
UserDisplayName: user8714896
ThreadText: # Concrete example of latent variables and observables plugged into the Bayes' rule
  
  (Asked by: username_0 on 2019-11-21)
  
  In the context of the variational auto-encoder, can someone give me a concrete example of the application of the Bayes' rule
  
  $$p\_{\theta}(z|x)=\frac{p\_{\theta}(x|z)p(z)}{p(x)}$$
  
  for a given latent variable and observable?
  
  I understand with VAE's we're essentially getting an approximation to $p\_{\theta}(z|x)$ that models the distribution that we think approximates the latent variables, but I need a concrete example to really understand it.
  
  **Tags:** convolutional-neural-networks, autoencoders, variational-autoencoder
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-11-22. Score: 4 | Accepted answer
  
  Let's assume the probability distributions are Gaussian (or normal) distributions. In other words, in the Bayes' rule
  
  \begin{align}
  p(z|x)=\frac{p(x|z)p(z)}{p(x)}
  \tag{1}\label{1}
  \end{align}
  
  The posterior $p(z|x)$, the likelihood $p(x|z)$, the prior $p(z)$ and the evidence (or marginal) $p(x)$ are Gaussian distributions. You can assume this because [Gaussian distributions are closed under conditioning and marginalization](https://distill.pub/2019/visual-exploration-gaussian-processes/#Multivariate).
  
  For simplicity, let's further assume that they are [**univariate** Gaussian distributions](https://en.wikipedia.org/wiki/Normal_distribution). Given that the Gaussian distribution is a *continuous probability distribution*, it has an associated *probability density function* (rather than a probability mass function, which is associated with discrete probability distributions, such as the [Bernoulli distribution](https://en.wikipedia.org/wiki/Bernoulli_distribution)). The probability density function of the Gaussian distribution is
  
  \begin{align}
  f(x \mid \mu, \sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2} } e^{ -\frac{(x-\mu)^2}{2\sigma^2} } \tag{2}\label{2}
  \end{align}
  
  where $\mu$ and $\sigma^2$ are respectively the mean and variance of the Gaussian distribution and $x$ is a variable (similarly to a variable $x$ in any mathematical function $f(x)$). So, given a concrete value for $x$, for example, $x=1$, then $f(x=1, \mu, \sigma^2)$ is a so-called **density** value (rather than a probability, which a probability mass function returns, given an input). For example, let's assume that the mean $\mu=0$ and the variance $\sigma^2 = 1$, then, for $x=1$, the density will be
  
  $$
  f(1 \mid 0, 1) = \frac{1}{\sqrt{2\pi} } e^{ -\frac{1}{2} }
  $$
  
  So, to obtain the concrete density value, I've just replaced the concrete values of $x$, $\mu$ and $\sigma^2$ in equation \ref{2}.
  
  To calculate the posterior $p(z|x)$ in equation \ref{1}, you just need to replace the likelihood $p(x|z)$, the prior $p(z)$ and the evidence $p(x)$ with the Gaussian probability density shown in equation \ref{2}, so you will have
  
  \begin{align}
  p(z|x)=\frac{f\_{X\mid Z}(x \mid \mu\_{X\mid Z}, \sigma^2\_{X\mid Z}, z) f\_{Z}(z \mid \mu\_{Z}, \sigma^2\_{Z})}{f\_{X}(x \mid \mu\_{X}, \sigma^2\_{X})}
  \tag{3}\label{3}
  \end{align}
  
  I've explicitly added a subscript to the means and variances of each probability density, given that, for example, the mean of the probability density $f\_{X\mid Z}$ might be different than the mean of the probability density $f\_{Z}$ or $f\_{X}$, etc. So, to get the actual density value (a real number) that represents $p(z|x)$, you just need to replace $f\_{X\mid Z}$, $f\_{Z}$ and $f\_{X}$ with the definition of the Gaussian density function in \ref{2} with their actual mean and variance values. I'll let you do this, given that this is really just a matter of picking up a concrete value for the means and variances and doing some algebra.
  
  If you assume the posterior, the likelihood, the prior or the evidence to have a different distribution, you will do the same thing, but using the probability density or mass function of your chosen distribution.
  
  In the context of the variational auto-encoder, you will be learning the mean and variance of the distribution, so the mean and variance will not be fixed, but they will be the parameters that you want to find. However, this does not change the way you apply the Bayes' rule.
  
  
  ---
  
  