Answers.Body: <h2>Short answer</h2>
  <p>The VAE is <strong>not</strong> the Auto-Encoding Variational Bayes (AEVB) algorithm or an instance of it.</p>
  <p>The VAE is a machine learning <strong>model</strong>. More specifically, it's a probabilistic auto-encoder, where the probabilistic encoder (aka recognition model) and decoder (aka generative model) are represented as neural networks. (Here, &quot;probabilistic&quot; means that they produce/represent a probability distribution).</p>
  <p>The AEVB algorithm is a <strong>learning/inference algorithm</strong> that can be used to find the parameters of the VAE, i.e. to perform approximate variational inference in the following directed graphical model (figure 1 of <a href="https://arxiv.org/pdf/1312.6114.pdf" rel="nofollow noreferrer">the VAE paper</a>). (Note that &quot;inference&quot; in Bayesian statistics has a specific meaning, i.e. the computation of the posterior using Bayes rule)</p>
  <p><a href="https://i.sstatic.net/XrdRC.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/XrdRC.png" alt="enter image description here" /></a></p>
  <h2>Long answer</h2>
  <h3>The AEVB <em>algorithm</em></h3>
  <p>The Auto-Encoding Variational Bayes (AEVB) is the algorithm used to find the parameters <span class="math-container">$\boldsymbol{\theta}$</span> and <span class="math-container">$\boldsymbol{\phi}$</span>, as you can conclude by reading its pseudocode given in the paper.</p>
  <p><a href="https://i.sstatic.net/rZLX8.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/rZLX8.png" alt="enter image description here" /></a></p>
  <p>The AEVB algorithm <strong>stochastically</strong> estimates the gradient of the objective function, i.e. the <strong>Evidence Lower BOund (ELBO)</strong> (<a href="https://arxiv.org/pdf/1312.6114.pdf#page=3" rel="nofollow noreferrer">equation 3</a>). They propose 2 estimators (equations 6 and 7), of the ELBO (not its gradient, which is computed automatically by back-propagation; note that stochastic estimation of the gradient of the objective function is very common in deep learning: see <a href="https://ai.stackexchange.com/a/20380/2444">this</a>). Moreover, the AEVB algorithm also uses the <strong>re-parametrization trick</strong> (i.e. it samples from <span class="math-container">$p(\boldsymbol{\epsilon})$</span> to avoid high variance).</p>
  <h3>The VAE <em>model</em></h3>
  <p>The VAE is the model that you create when you use neural networks to represent the encoder (aka recognition model) and decoder (aka generative model), which are parametrized by <span class="math-container">$\boldsymbol{\phi}$</span> and <span class="math-container">$\boldsymbol{\theta}$</span>, respectively. So, <strong>the VAE is not the AEVB algorithm</strong>. You use the latter to find the parameters of the VAE.</p>
  <p>The authors write in the same paragraph that you are quoting</p>
  <blockquote>
  <p>When a neural network is used for the <strong>recognition model</strong>, we arrive at the variational auto-encoder.</p>
  </blockquote>
  <p>In the appendix, they also write</p>
  <blockquote>
  <p>In variational auto-encoders, neural networks are used as probabilistic encoders and decoders. There are many possible choices of encoders and decoders, depending on the type of data and model. In our example we used relatively simple neural networks, namely multi-layered perceptrons (MLPs). For the encoder we used a MLP with Gaussian output, while for the decoder we used MLPs with either Gaussian or Bernoulli outputs, depending on the type of data.</p>
  </blockquote>
  <h3>Variational and generative parameters</h3>
  <p>Ok, but what are these parameters <span class="math-container">$\boldsymbol{\phi}$</span> and <span class="math-container">$\boldsymbol{\theta}$</span>?</p>
  <ul>
  <li><p><span class="math-container">$\boldsymbol{\phi}$</span> are the parameters of the <strong>variational distribution</strong> <span class="math-container">$q_{\boldsymbol{\phi}}(\mathbf{z} \mid \mathbf{x})$</span>, which is an approximation of the (usually intractable) posterior <span class="math-container">$p_{\boldsymbol{\theta}}(\mathbf{z} \mid \mathbf{x}) = p_{\boldsymbol{\theta}}(\mathbf{z}) p_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z})/p_{\boldsymbol{\theta}}(\mathbf{x})$</span>; so, in the case of the VAE, these are the weights of the encoder</p>
  </li>
  <li><p><span class="math-container">$\boldsymbol{\theta}$</span> are the parameters of the <strong>generative model</strong> <span class="math-container">$p_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z})$</span>; so, in the case of the VAE, these are the weights (and biases) of the decoder neural network</p>
  </li>
  </ul>
  <h3>Recap</h3>
  <p>So, to recap, the AEVB is the training/learning/inference algorithm to find <span class="math-container">$\boldsymbol{\phi}$</span> and <span class="math-container">$\boldsymbol{\theta}$</span>. The AVB stochastically estimates the ELBO (and, consequently, its gradient) and uses the re-parametrization trick. If you use a neural network to represent the encoder and the decoder, you get the VAE, which is not the same thing as the AEVB, the algorithm to find the parameters of the neural networks (i.e. the encoder and decoder).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-29T18:50:16.947Z
Answers.Text: Not sure if I should ask this in a separate question, but is the SGVB estimator the estimator for  the lower bound?   The paper states this is the 2nd version of the SGVB estimator :
Answers.UserDisplayName: a12345
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-29T18:51:41.993Z
Answers.Text: $$\begin{aligned}
  &\widetilde{\mathcal{L}}^{B}\left(\boldsymbol{\theta}, \boldsymbol{\phi} ; \mathbf{x}^{(i)}\right)=-D_{K L}\left(q_{\phi}\left(\mathbf{z} \mid \mathbf{x}^{(i)}\right) \| p_{\theta}(\mathbf{z})\right)+\frac{1}{L} \sum_{l=1}^{L}\left(\log p_{\theta}\left(\mathbf{x}^{(i)} \mid \mathbf{z}^{(i, l)}\right)\right) \\
  &\text { where } \mathbf{z}^{(i, l)}=g_{\phi}\left(\epsilon^{(i, l)}, \mathbf{x}^{(i)}\right) \text { and } \epsilon^{(l)} \sim p(\epsilon)
  \end{aligned}$$
Answers.UserDisplayName: a12345
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-29T19:53:04.927Z
Answers.Text: @a12345 Yes, it's an estimator of the ELBO objective function, not its gradient (which is computed _automatically_ with back-propagation). In the paper/pseudocode, the gradient of the estimator is denoted by $\mathbf{g}$. You can see there they use $$\nabla_{\boldsymbol{\theta}, \boldsymbol{\phi}} \widetilde{\mathcal{L}}^{M}\left(\boldsymbol{\theta}, \boldsymbol{\phi} ; \mathbf{X}^{M}, \boldsymbol{\epsilon}\right)$$. Anyway, you can ask this in a separate question too, and I will provide this answer there. Maybe this question can be useful for future readers.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-29T12:16:38.360Z
Answers.LastActivityDate: 2021-10-30T13:09:47.600Z
Answers.LastEditDate: 2021-10-30T13:09:47.600Z
Answers.UserDisplayName: nbro
Body: <p>I am familiar with the variational autoencoder, but not totally clear on what exactly the AEVB is.</p>
  <p>In <a href="https://arxiv.org/abs/1312.6114" rel="nofollow noreferrer">the original VAE paper</a> (by Kingma and Welling), he uses both the terms variational autoencoder and autoencoding variational Bayes.</p>
  <blockquote>
  <p>For the case of an i.i.d. dataset and continuous latent variables per datapoint, we propose the Auto-Encoding VB (AEVB) algorithm. In the AEVB algorithm, we make inference and learning especially efficient by using the SGVB estimator to optimize a recognition model that allows us to perform very efficient approximate posterior inference using simple ancestral sampling, which in turn allows us to efficiently learn the model parameters, without the need of expensive iterative inference schemes (such as MCMC) per datapoint.</p>
  </blockquote>
  <p>And then in section 2, the paper talks about the SGVB estimator and the AEVB algorithm.</p>
  <p>Then in section 3, it says that the VAE is an example.</p>
  <p>So, is the VAE a special application of AEVB?</p>
  
Tags: <comparison><terminology><papers><variational-autoencoder><aevb-algorithm>
Title: How is the VAE related to the Autoencoding Variational Bayes (AEVB) algorithm?
UserDisplayName: a12345
ThreadText: # How is the VAE related to the Autoencoding Variational Bayes (AEVB) algorithm?
  
  (Asked by: username_0 on 2021-10-28)
  
  I am familiar with the variational autoencoder, but not totally clear on what exactly the AEVB is.
  
  In [the original VAE paper](https://arxiv.org/abs/1312.6114) (by Kingma and Welling), he uses both the terms variational autoencoder and autoencoding variational Bayes.
  
  > For the case of an i.i.d. dataset and continuous latent variables per datapoint, we propose the Auto-Encoding VB (AEVB) algorithm. In the AEVB algorithm, we make inference and learning especially efficient by using the SGVB estimator to optimize a recognition model that allows us to perform very efficient approximate posterior inference using simple ancestral sampling, which in turn allows us to efficiently learn the model parameters, without the need of expensive iterative inference schemes (such as MCMC) per datapoint.
  
  And then in section 2, the paper talks about the SGVB estimator and the AEVB algorithm.
  
  Then in section 3, it says that the VAE is an example.
  
  So, is the VAE a special application of AEVB?
  
  **Tags:** comparison, terminology, papers, variational-autoencoder, aevb-algorithm
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-10-29. Score: 2 | Accepted answer
  
  ## Short answer
  
  The VAE is **not** the Auto-Encoding Variational Bayes (AEVB) algorithm or an instance of it.
  
  The VAE is a machine learning **model**. More specifically, it's a probabilistic auto-encoder, where the probabilistic encoder (aka recognition model) and decoder (aka generative model) are represented as neural networks. (Here, "probabilistic" means that they produce/represent a probability distribution).
  
  The AEVB algorithm is a **learning/inference algorithm** that can be used to find the parameters of the VAE, i.e. to perform approximate variational inference in the following directed graphical model (figure 1 of [the VAE paper](https://arxiv.org/pdf/1312.6114.pdf)). (Note that "inference" in Bayesian statistics has a specific meaning, i.e. the computation of the posterior using Bayes rule)
  
  [![enter image description here](https://i.sstatic.net/XrdRC.png)](https://i.sstatic.net/XrdRC.png)
  
  ## Long answer
  
  ### The AEVB *algorithm*
  
  The Auto-Encoding Variational Bayes (AEVB) is the algorithm used to find the parameters $\boldsymbol{\theta}$ and $\boldsymbol{\phi}$, as you can conclude by reading its pseudocode given in the paper.
  
  [![enter image description here](https://i.sstatic.net/rZLX8.png)](https://i.sstatic.net/rZLX8.png)
  
  The AEVB algorithm **stochastically** estimates the gradient of the objective function, i.e. the **Evidence Lower BOund (ELBO)** ([equation 3](https://arxiv.org/pdf/1312.6114.pdf#page=3)). They propose 2 estimators (equations 6 and 7), of the ELBO (not its gradient, which is computed automatically by back-propagation; note that stochastic estimation of the gradient of the objective function is very common in deep learning: see [this](https://ai.stackexchange.com/a/20380/2444)). Moreover, the AEVB algorithm also uses the **re-parametrization trick** (i.e. it samples from $p(\boldsymbol{\epsilon})$ to avoid high variance).
  
  ### The VAE *model*
  
  The VAE is the model that you create when you use neural networks to represent the encoder (aka recognition model) and decoder (aka generative model), which are parametrized by $\boldsymbol{\phi}$ and $\boldsymbol{\theta}$, respectively. So, **the VAE is not the AEVB algorithm**. You use the latter to find the parameters of the VAE.
  
  The authors write in the same paragraph that you are quoting
  
  > When a neural network is used for the **recognition model**, we arrive at the variational auto-encoder.
  
  In the appendix, they also write
  
  > In variational auto-encoders, neural networks are used as probabilistic encoders and decoders. There are many possible choices of encoders and decoders, depending on the type of data and model. In our example we used relatively simple neural networks, namely multi-layered perceptrons (MLPs). For the encoder we used a MLP with Gaussian output, while for the decoder we used MLPs with either Gaussian or Bernoulli outputs, depending on the type of data.
  
  ### Variational and generative parameters
  
  Ok, but what are these parameters $\boldsymbol{\phi}$ and $\boldsymbol{\theta}$?
  
  * $\boldsymbol{\phi}$ are the parameters of the **variational distribution** $q\_{\boldsymbol{\phi}}(\mathbf{z} \mid \mathbf{x})$, which is an approximation of the (usually intractable) posterior $p\_{\boldsymbol{\theta}}(\mathbf{z} \mid \mathbf{x}) = p\_{\boldsymbol{\theta}}(\mathbf{z}) p\_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z})/p\_{\boldsymbol{\theta}}(\mathbf{x})$; so, in the case of the VAE, these are the weights of the encoder
  * $\boldsymbol{\theta}$ are the parameters of the **generative model** $p\_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z})$; so, in the case of the VAE, these are the weights (and biases) of the decoder neural network
  
  ### Recap
  
  So, to recap, the AEVB is the training/learning/inference algorithm to find $\boldsymbol{\phi}$ and $\boldsymbol{\theta}$. The AVB stochastically estimates the ELBO (and, consequently, its gradient) and uses the re-parametrization trick. If you use a neural network to represent the encoder and the decoder, you get the VAE, which is not the same thing as the AEVB, the algorithm to find the parameters of the neural networks (i.e. the encoder and decoder).
  
  > **Comments:**
  > **username_0**: Not sure if I should ask this in a separate question, but is the SGVB estimator the estimator for  the lower bound?   The paper states this is the 2nd version of the SGVB estimator :  
  > **username_0**: $$\begin{aligned} &\widetilde{\mathcal{L}}^{B}\left(\boldsymbol{\theta}, \boldsymbol{\phi} ; \mathbf{x}^{(i)}\right)=-D_{K L}\left(q_{\phi}\left(\mathbf{z} \mid \mathbf{x}^{(i)}\right) \| p_{\theta}(\mathbf{z})\right)+\frac{1}{L} \sum_{l=1}^{L}\left(\log p_{\theta}\left(\mathbf{x}^{(i)} \mid \mathbf{z}^{(i, l)}\right)\right) \\ &\text { where } \mathbf{z}^{(i, l)}=g_{\phi}\left(\epsilon^{(i, l)}, \mathbf{x}^{(i)}\right) \text { and } \epsilon^{(l)} \sim p(\epsilon) \end{aligned}$$  
  > **username_1**: @a12345 Yes, it's an estimator of the ELBO objective function, not its gradient (which is computed _automatically_ with back-propagation). In the paper/pseudocode, the gradient of the estimator is denoted by $\mathbf{g}$. You can see there they use $$\nabla_{\boldsymbol{\theta}, \boldsymbol{\phi}} \widetilde{\mathcal{L}}^{M}\left(\boldsymbol{\theta}, \boldsymbol{\phi} ; \mathbf{X}^{M}, \boldsymbol{\epsilon}\right)$$. Anyway, you can ask this in a separate question too, and I will provide this answer there. Maybe this question can be useful for future readers.  
  
  ---
  
  