Answers.Body: <p>As already mentioned in the comment, the reason, why the does the backpropagation still work is the <a href="https://stats.stackexchange.com/questions/199605/how-does-the-reparameterization-trick-for-vaes-work-and-why-is-it-important">Reparametrization Trick</a>.</p>
  <p>For variational autoencoder (VAE) neural networks to be learned predict parameters of the random distribution - the mean <span class="math-container">$\mu_{\theta} (x)$</span> and the variance <span class="math-container">$\sigma_{\phi} (x)$</span> for the case on normal distribution. Here <span class="math-container">$\theta$</span> and <span class="math-container">$\phi$</span> are parameters, that can be learned during the backprogation.</p>
  <p>At each sampling step, one generates random number <span class="math-container">$\varepsilon$</span> from the normal distribution with zero mean and unit variance <span class="math-container">$\mathcal{N}(0, 1)$</span> and then multiplies this number by <span class="math-container">$\sigma_{\phi} (x)$</span> and adds <span class="math-container">$\mu_{\theta} (x)$</span>:
  <span class="math-container">$$
  \mu_{\theta} (x) + \varepsilon \sigma_{\phi} (x) 
  $$</span>
  The output of this operation has a normal distribution <span class="math-container">$\mathcal{N}(\mu_{\theta} (x), \sigma_{\phi} (x))$</span>.
  You do not have to backpropagate through the epsilon, and this number is just multiplied by  <span class="math-container">$\sigma_{\phi} (x) $</span>.</p>
  <p>The whole procedure is nicely differentiable with respect to the <span class="math-container">$\theta$</span> and <span class="math-container">$\phi$</span>.</p>
  <p>However, note , that the possibility to apply this trick is allowed due to the nice property of normal distribution:
  <span class="math-container">$$
  \mathcal{N}(a, b) \sim a + b \mathcal{N}(0, 1)
  $$</span>
  Arbitrary distribution doesn't satisfy this property.</p>
  <p>In more general case, you would need to apply some other trick to propagate the gradients.</p>
  <p>Simple strategy is the  <a href="https://arxiv.org/abs/1308.3432" rel="nofollow noreferrer">Straight through estimator</a>, when one copies the gradient from the output layer to the input through non-differentialbe operation. Or more accurate <a href="https://arxiv.org/abs/1611.01144" rel="nofollow noreferrer">Gumbel-Softmax softmax</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-19T08:11:14.727Z
Answers.LastActivityDate: 2021-12-19T08:11:14.727Z
Answers.UserDisplayName: spiridon_the_sun_rotator
Answers.Body: <p>Sampling per-se is non-differentiable in some point of view, but you can use the reparametrisation in some special scenario.</p>
  <p>Differentiation, mathematically, is the act of taking derivatives of functions wrt variables. Sampling is the act of generating a set of elements given some parameters, and differentiating a function that outputs a set is, to my knowledge, not well discussed in the setting of machine learning. More often, sampling is followed by an expectation operator, where you takes average of a function across your samples. And this expectation operator, is differentiable.</p>
  <p><span class="math-container">$$E_\pi(X(s))={1\over N} \sum_{i=1,\{s_i\}\sim\pi}^N X(s_i)=\sum_s X(s)P_\pi(s)$$</span></p>
  <p>In fact, backpropagating through a sampling operation has a well known name -- it's called &quot;policy gradient&quot; in reinforcement learning, where you study how to take samples (known as policy) so as to minimise error (known as maximising the expected reward).</p>
  <p>Not sure why this connection has not been well studied yet... maybe NN people just love a fully deterministic function approximator.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-18T09:19:49.933Z
Answers.Text: You don't have to leave the "notice" in your new answer. Moderators can convert answer-like comments to comments or even delete them. This is not weird here because we expect users to provide answers with details and some users might have written a comment as an answer because they don't know how the site works or maybe they don't have the reputation to comment. Having said that, you can post another answer, as you did, once you have written a more detailed one.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-18T10:11:38.547Z
Answers.Text: I just wanted to say, that I was not happy, and the moderator is now removing my notice. My emotion was not acknowledged -- it was REMOVED. I don't feel my voice respected anymore. People are not machines.
Answers.UserDisplayName: shouldsee
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-18T11:17:15.317Z
Answers.Text: I removed the noticed, and I am currently not a mod. So it was not a moderator that removed the notice. People that have more than 2k rep here can edit posts without confirmation, like I did. Our site's model discourages people to leave unnecessary details in the questions or answers, with respect to the question being asked. Things like "Thanks!" or "Have a nice day" are discouraged in questions or answers. It's not because we're nice, but because we want to create an archive of questions and answers where people can easily find their answer without losing time reading unnecessary details.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-18T11:20:25.167Z
Answers.Text: I understand your feelings, but you have to understand that this site is not meant for people to share their opinions, but to give answers to questions. Anyway, thanks for trying to contributing to site. If you haven't done it yet, you might want to have a look at our site [on-topic page](https://ai.stackexchange.com/help/on-topic).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-18T11:34:30.773Z
Answers.Text: Ok thanks. I am happy to see we are actually trying to communicate. I agree with the general intention that to keep the site well-structured, but editing is a strong privilege and the mechanism could definitely be improved to balance between motivating contribution while keep the quality high.
Answers.UserDisplayName: shouldsee
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-18T04:12:10.147Z
Answers.LastActivityDate: 2022-03-18T09:18:38.587Z
Answers.LastEditDate: 2022-03-18T09:18:38.587Z
Answers.UserDisplayName: shouldsee
Answers.Body: <p>You do not backpropagate with respect to <span class="math-container">$\epsilon$</span>, which is the random sample or random variable (depending on how you look at it). You backpropagate with respect to the mean <span class="math-container">$\mu$</span> and variance <span class="math-container">$\sigma$</span> of the latent Gaussian (the variational distribution). Note that, although <span class="math-container">$z$</span> is a random sample (and not just a sample), because it's computed as a function of <span class="math-container">$\epsilon$</span> (a random sample, once it's sampled from e.g. <span class="math-container">$\mathcal{N}(0, 1)$</span>), <span class="math-container">$\mu$</span> and <span class="math-container">$\sigma$</span> are not: these are learnable parameters and are deterministic.</p>
  <p>Having said this, note that we use the <a href="https://arxiv.org/pdf/1312.6114.pdf" rel="nofollow noreferrer"><strong>reparametrization trick</strong></a> in the VAE, so we compute the random sample as <span class="math-container">$z = g_\phi(\epsilon, x))$</span>, where <span class="math-container">$g_\phi$</span> is a deterministic function (encoder) parametrized by <span class="math-container">$\phi$</span> (the weights of the neural network that represents the encoder). In case the random variable <span class="math-container">$z \sim \mathcal{N}(\mu, \sigma^2)$</span>, then we can express the random variable (so also the random sample) as follows <span class="math-container">$z=g_\phi(\epsilon, x)) = \mu+\sigma \epsilon$</span>. So, as you can see from the code, we sample <span class="math-container">$\epsilon$</span> from some prior <span class="math-container">$p(\epsilon)$</span> (e.g. <span class="math-container">$\mathcal{N}(0, 1)$</span>), then we compute <span class="math-container">$z$</span> deterministically.</p>
  <p>Why is this <strong>reparametrization trick</strong> useful? The authors of the VAE paper explain it.</p>
  <blockquote>
  <p>This reparameterization is useful for our case since it can be used to rewrite an expectation w.r.t <span class="math-container">$q_{\phi}(\mathbf{z} \mid \mathbf{x})$</span> such that the Monte Carlo estimate of the expectation is differentiable w.r.t. <span class="math-container">$\phi$</span>. A proof is as follows. Given the deterministic mapping <span class="math-container">$\mathbf{z}=g_{\phi}(\boldsymbol{\epsilon}, \mathbf{x})$</span> we know that <span class="math-container">$q_{\phi}(\mathbf{z} \mid \mathbf{x}) \prod_{i} d z_{i}=$</span> <span class="math-container">$p(\boldsymbol{\epsilon}) \prod_{i} d \epsilon_{i}$</span>. Therefore <span class="math-container">$^{1}, \int q_{\boldsymbol{\phi}}(\mathbf{z} \mid \mathbf{x}) f(\mathbf{z}) d \mathbf{z}=\int p(\boldsymbol{\epsilon}) f(\mathbf{z}) d \boldsymbol{\epsilon}=\int p(\boldsymbol{\epsilon}) f\left(g_{\boldsymbol{\phi}}(\boldsymbol{\epsilon}, \mathbf{x})\right) d \boldsymbol{\epsilon}$</span>. It follows that a differentiable estimator can be constructed: <span class="math-container">$\int q_{\phi}(\mathbf{z} \mid \mathbf{x}) f(\mathbf{z}) d \mathbf{z} \simeq \frac{1}{L} \sum_{l=1}^{L} f\left(g_{\phi}\left(\mathbf{x}, \boldsymbol{\epsilon}^{(l)}\right)\right)$</span> where <span class="math-container">$\boldsymbol{\epsilon}^{(l)} \sim p(\boldsymbol{\epsilon}).$</span></p>
  </blockquote>
  <p>Note that <span class="math-container">$\mathbb{E}_{q_{\boldsymbol{\phi}}(\mathbf{z} \mid \mathbf{x})} \left[ f(\mathbf{z}) \right] = \int q_{\boldsymbol{\phi}}(\mathbf{z} \mid \mathbf{x}) f(\mathbf{z}) d \mathbf{z} = \int p(\boldsymbol{\epsilon}) f\left(g_{\boldsymbol{\phi}}(\boldsymbol{\epsilon}, \mathbf{x})\right) d \boldsymbol{\epsilon} = \mathbb{E}_{p(\boldsymbol{\epsilon})} \left[ f\left(g_{\boldsymbol{\phi}}(\boldsymbol{\epsilon}, \mathbf{x})\right) \right]$</span>, which can be <strong>estimated</strong> with <span class="math-container">$\frac{1}{L} \sum_{l=1}^{L} f\left(g_{\phi}\left(\mathbf{x}, \boldsymbol{\epsilon}^{(l)}\right)\right)$</span> where <span class="math-container">$\boldsymbol{\epsilon}^{(l)} \sim p(\boldsymbol{\epsilon})$</span>. In other words, you can sample <span class="math-container">$L$</span> <span class="math-container">$z$</span> in the way we did in order to estimate <span class="math-container">$\mathbb{E}_{\color{blue}{q_{\boldsymbol{\phi}}(\mathbf{z} \mid \mathbf{x})}} \left[ \color{red}{f(\mathbf{z})} \right]$</span> (this are Monte Carlo estimates of the expectation).</p>
  <p>In the case of the VAE, we want to optimize the ELBO, which is the following objective function</p>
  <p><span class="math-container">$$\mathcal{L}\left(\boldsymbol{\theta}, \boldsymbol{\phi} ; \mathbf{x}^{(i)}\right)=\underbrace{-D_{K L}\left(q_{\boldsymbol{\phi}}\left(\mathbf{z} \mid \mathbf{x}^{(i)}\right) \| p_{\boldsymbol{\theta}}(\mathbf{z})\right)}_{\text{KL divergence}}+ \underbrace{\mathbb{E}_{\color{blue}{q_{\phi}\left(\mathbf{z} \mid \mathbf{x}^{(i)}\right)}}\left[\color{red}{\log p_{\boldsymbol{\theta}}\left(\mathbf{x}^{(i)} \mid \mathbf{z}\right)}\right]}_{\text{likelihood}}$$</span></p>
  <p>which we can estimate with Monte Carlo estimates of the second term (the <strong>likelihood term</strong>)</p>
  <p><span class="math-container">$$
  \widetilde{\mathcal{L}}^{B}\left(\boldsymbol{\theta}, \boldsymbol{\phi} ; \mathbf{x}^{(i)}\right)=-D_{K L}\left(q_{\boldsymbol{\phi}}\left(\mathbf{z} \mid \mathbf{x}^{(i)}\right) \| p_{\boldsymbol{\theta}}(\mathbf{z})\right)+ \underbrace{\frac{1}{L} \sum_{l=1}^{L}\left(\log p_{\boldsymbol{\theta}}\left(\mathbf{x}^{(i)} \mid \mathbf{z}^{(i, l)}\right)\right)}_{\text{likelihood}}
  $$</span>
  where <span class="math-container">$$
  \text { where } \quad \mathbf{z}^{(i, l)}=g_{\phi}\left(\boldsymbol{\epsilon}^{(i, l)}, \mathbf{x}^{(i)}\right) \text { and } \boldsymbol{\epsilon}^{(l)} \sim p(\boldsymbol{\epsilon})
  $$</span></p>
  <p>Here, the likelihood is computed with the neural network, for example, in practice, you use the cross-entropy of the output of your decoder, which gets as input the input to the decoder, hence <span class="math-container">$z$</span>.</p>
  <p>You can ignore the KL divergence now because, in the case of Gaussians, <a href="https://ai.stackexchange.com/a/26408/2444">it can be computed analytically</a>.</p>
  <p>Now, what if we didn't use the reparametrization trick? Could we still backpropagate with respect to <span class="math-container">$\phi$</span>? The authors of the VAE write</p>
  <blockquote>
  <p>The usual (naïve) Monte Carlo gradient estimator for this type of problem is: <span class="math-container">$\nabla_{\phi} \mathbb{E}_{q_{\phi}(\mathbf{z})}[f(\mathbf{z})]=\mathbb{E}_{q_{\phi}(\mathbf{z})}\left[f(\mathbf{z}) \nabla_{q_{\phi}(\mathbf{z})} \log q_{\phi}(\mathbf{z})\right] \simeq \frac{1}{L} \sum_{l=1}^{L} f(\mathbf{z}) \nabla_{q_{\phi}\left(\mathbf{z}^{(l)}\right)} \log q_{\phi}\left(\mathbf{z}^{(l)}\right)$</span> where <span class="math-container">$\mathbf{z}^{(l)} \sim q_{\phi}\left(\mathbf{z} \mid \mathbf{x}^{(i)}\right) .$</span> This gradient estimator exhibits exhibits very high variance (see e.g. <a href="https://arxiv.org/pdf/1206.6430.pdf" rel="nofollow noreferrer">[BJP12]</a>)</p>
  </blockquote>
  <p>So, the answer is <strong>yes</strong>, as opposed to what many people might say around, but with another estimator (which may remind you of some equations you have seen in reinforcement learning related to <a href="http://incompleteideas.net/book/RLbook2020.pdf#page=348" rel="nofollow noreferrer">REINFORCE</a>, if you are familiar with this), i.e. <span class="math-container">$$\frac{1}{L} \sum_{l=1}^{L} f(\mathbf{z}) \nabla_{q_{\phi}\left(\mathbf{z}^{(l)}\right)} \log q_{\phi}\left(\mathbf{z}^{(l)}\right) \tag{1}\label{1},$$</span> which has high variance.</p>
  <p>So, in the end, the reparametrization trick can be viewed as a <a href="http://proceedings.mlr.press/v89/xu19a/xu19a.pdf" rel="nofollow noreferrer">variance reduction technique</a>. There are others, like <a href="https://arxiv.org/pdf/1206.6430.pdf" rel="nofollow noreferrer"><strong>control variates</strong></a> or <a href="https://arxiv.org/abs/1803.04386" rel="nofollow noreferrer"><strong>Flipout</strong></a> (<a href="https://github.com/tensorflow/probability/blob/main/tensorflow_probability/examples/bayesian_neural_network.py" rel="nofollow noreferrer">used e.g. in the context of Bayesian neural networks</a>).</p>
  <p>The first thing to note about \ref{1} is that we do not need to take the derivative with respect to <span class="math-container">$f$</span>. The second thing is that <span class="math-container">$\nabla_{q_{\phi}\left(\mathbf{z}^{(l)}\right)} \log q_{\phi}\left(\mathbf{z}^{(l)}\right)$</span> is the <a href="https://jmlr.org/papers/volume21/19-346/19-346.pdf" rel="nofollow noreferrer"><strong>score function</strong></a>.</p>
  <p>Now, don't ask me how to calculate this gradient <span class="math-container">$\nabla_{q_{\phi}\left(\mathbf{z}^{(l)}\right)} \log q_{\phi}\left(\mathbf{z}^{(l)}\right)$</span> (because I am bad at math). However, note that <span class="math-container">$
  \mathbf{z}^{(l)} \sim q_{\phi}\left(\mathbf{z} \mid \mathbf{x}^{(i)}\right)
  $</span>, so we treat <span class="math-container">$\mathbf{z}^{(l)}$</span> as a fixed random sample. By the way, I am not sure whether they used <span class="math-container">$q_{\phi}\left(\mathbf{z} \mid \mathbf{x}^{(i)}\right)$</span> differently from <span class="math-container">$q_{\phi}\left(\mathbf{z} \right)$</span>. I don't think so. I think they are the same and I think we can still back-propagate with respect to <span class="math-container">$\phi$</span>, even if we use it to sample <span class="math-container">$\mathbf{z}^{(i)}$</span>, because, once this latter is sampled, it can be treated as fixed (random) sample.</p>
  <p>I also note that I think they meant <span class="math-container">$\frac{1}{L} \sum_{l=1}^{L} f(\mathbf{z}^{(l)}) \nabla_{q_{\phi}\left(\mathbf{z}^{(l)}\right)} \log q_{\phi}\left(\mathbf{z}^{(l)}\right)$</span>, i.e. they forget to use <span class="math-container">$\mathbf{z}^{(l)}$</span> rather than just <span class="math-container">$\mathbf{z}$</span> as input for <span class="math-container">$f$</span> in the Monte Carto estimate. You can see in section 3 of <a href="https://arxiv.org/pdf/1206.6430.pdf" rel="nofollow noreferrer">[BJP12]</a> and section 4.2. of <a href="https://jmlr.org/papers/volume21/19-346/19-346.pdf" rel="nofollow noreferrer">[1]</a> they do it like this, and it makes sense. So, the VAE paper has sloppy stuff in there.</p>
  <p>My intuition of why this has high variance is because you sample something according to the variational distribution, but then you update this variational distribution, and you continuously do this. However, I am not sure this is the right intuition. In <a href="https://arxiv.org/pdf/1206.6430.pdf" rel="nofollow noreferrer">[BJP12]</a>, they say (in section 4) this MC estimate has high variance and that you need a lot of samples <span class="math-container">$\mathbf{x}$</span>. I don't know exactly why this is the case because I didn't fully read this paper yet.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-17T16:58:53.727Z
Answers.Text: Nice. Worth adding LHS to eq1.
Answers.UserDisplayName: shouldsee
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-19T09:30:14.640Z
Answers.LastActivityDate: 2022-06-04T15:08:08.633Z
Answers.LastEditDate: 2022-06-04T15:08:08.633Z
Answers.UserDisplayName: nbro
Body: <p>Implementations of variational autoencoders that I've looked at all include a sampling layer as the last layer of the encoder block.  The encoder learns to generate a mean and standard deviation for each input, and samples from it to get the input's representation in latent space.  The decoder then attempts to decode this back out to match the inputs.</p>
  <p><strong>My question</strong>: How does backpropagation handle the random sampling step?</p>
  <p>Random sampling is not a deterministic function and doesn't have a derivative.  In order to train the encoder, gradient updates must somehow propagate back from the loss, through the sampling layer.</p>
  <p>I did my best to hunt for the source code for tensorflow's autodifferentiation of this function, but couldn't find it.  Here's an example of a keras implementation of that sampling step, from <a href="https://keras.io/examples/generative/vae/" rel="noreferrer">the keras docs</a>, in which <code>tf.keras.backend.random_normal</code> is used for the sampling.</p>
  <pre><code>class Sampling(layers.Layer):
      &quot;&quot;&quot;Uses (z_mean, z_log_var) to sample z, the vector encoding a digit.&quot;&quot;&quot;
  
      def call(self, inputs):
          z_mean, z_log_var = inputs
          batch = tf.shape(z_mean)[0]
          dim = tf.shape(z_mean)[1]
          epsilon = tf.keras.backend.random_normal(shape=(batch, dim))
          return z_mean + tf.exp(0.5 * z_log_var) * epsilon
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-19T01:37:59.557Z
Comments.Text: It is likely this will be done using the [reparameterisation trick](https://stats.stackexchange.com/questions/199605/how-does-the-reparameterization-trick-for-vaes-work-and-why-is-it-important)
Comments.UserDisplayName: David
Tags: <backpropagation><variational-autoencoder>
Title: How does backprop work through the random sampling layer in a variational autoencoder?
UserDisplayName: Luke Wolcott
ThreadText: # How does backprop work through the random sampling layer in a variational autoencoder?
  
  (Asked by: username_0 on 2021-12-18)
  
  Implementations of variational autoencoders that I've looked at all include a sampling layer as the last layer of the encoder block.  The encoder learns to generate a mean and standard deviation for each input, and samples from it to get the input's representation in latent space.  The decoder then attempts to decode this back out to match the inputs.
  My question: How does backpropagation handle the random sampling step?
  Random sampling is not a deterministic function and doesn't have a derivative.  In order to train the encoder, gradient updates must somehow propagate back from the loss, through the sampling layer.
  I did my best to hunt for the source code for tensorflow's autodifferentiation of this function, but couldn't find it.  Here's an example of a keras implementation of that sampling step, from the keras docs, in which tf.keras.backend.random_normal is used for the sampling.
  class Sampling(layers.Layer):
      """Uses (z_mean, z_log_var) to sample z, the vector encoding a digit."""
  
      def call(self, inputs):
          z_mean, z_log_var = inputs
          batch = tf.shape(z_mean)[0]
          dim = tf.shape(z_mean)[1]
          epsilon = tf.keras.backend.random_normal(shape=(batch, dim))
          return z_mean + tf.exp(0.5 * z_log_var) * epsilon
  
  **Tags:** backpropagation, variational-autoencoder
  > **Comments:**
  > **username_1**: It is likely this will be done using the [reparameterisation trick](https://stats.stackexchange.com/questions/199605/how-does-the-reparameterization-trick-for-vaes-work-and-why-is-it-important)  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_4 on 2021-12-19. Score: 4 | Accepted answer
  
  You do not backpropagate with respect to $\epsilon$, which is the random sample or random variable (depending on how you look at it). You backpropagate with respect to the mean $\mu$ and variance $\sigma$ of the latent Gaussian (the variational distribution). Note that, although $z$ is a random sample (and not just a sample), because it's computed as a function of $\epsilon$ (a random sample, once it's sampled from e.g. $\mathcal{N}(0, 1)$), $\mu$ and $\sigma$ are not: these are learnable parameters and are deterministic.
  
  Having said this, note that we use the [**reparametrization trick**](https://arxiv.org/pdf/1312.6114.pdf) in the VAE, so we compute the random sample as $z = g\_\phi(\epsilon, x))$, where $g\_\phi$ is a deterministic function (encoder) parametrized by $\phi$ (the weights of the neural network that represents the encoder). In case the random variable $z \sim \mathcal{N}(\mu, \sigma^2)$, then we can express the random variable (so also the random sample) as follows $z=g\_\phi(\epsilon, x)) = \mu+\sigma \epsilon$. So, as you can see from the code, we sample $\epsilon$ from some prior $p(\epsilon)$ (e.g. $\mathcal{N}(0, 1)$), then we compute $z$ deterministically.
  
  Why is this **reparametrization trick** useful? The authors of the VAE paper explain it.
  
  > This reparameterization is useful for our case since it can be used to rewrite an expectation w.r.t $q\_{\phi}(\mathbf{z} \mid \mathbf{x})$ such that the Monte Carlo estimate of the expectation is differentiable w.r.t. $\phi$. A proof is as follows. Given the deterministic mapping $\mathbf{z}=g\_{\phi}(\boldsymbol{\epsilon}, \mathbf{x})$ we know that $q\_{\phi}(\mathbf{z} \mid \mathbf{x}) \prod\_{i} d z\_{i}=$ $p(\boldsymbol{\epsilon}) \prod\_{i} d \epsilon\_{i}$. Therefore $^{1}, \int q\_{\boldsymbol{\phi}}(\mathbf{z} \mid \mathbf{x}) f(\mathbf{z}) d \mathbf{z}=\int p(\boldsymbol{\epsilon}) f(\mathbf{z}) d \boldsymbol{\epsilon}=\int p(\boldsymbol{\epsilon}) f\left(g\_{\boldsymbol{\phi}}(\boldsymbol{\epsilon}, \mathbf{x})\right) d \boldsymbol{\epsilon}$. It follows that a differentiable estimator can be constructed: $\int q\_{\phi}(\mathbf{z} \mid \mathbf{x}) f(\mathbf{z}) d \mathbf{z} \simeq \frac{1}{L} \sum\_{l=1}^{L} f\left(g\_{\phi}\left(\mathbf{x}, \boldsymbol{\epsilon}^{(l)}\right)\right)$ where $\boldsymbol{\epsilon}^{(l)} \sim p(\boldsymbol{\epsilon}).$
  
  Note that $\mathbb{E}\_{q\_{\boldsymbol{\phi}}(\mathbf{z} \mid \mathbf{x})} \left[ f(\mathbf{z}) \right] = \int q\_{\boldsymbol{\phi}}(\mathbf{z} \mid \mathbf{x}) f(\mathbf{z}) d \mathbf{z} = \int p(\boldsymbol{\epsilon}) f\left(g\_{\boldsymbol{\phi}}(\boldsymbol{\epsilon}, \mathbf{x})\right) d \boldsymbol{\epsilon} = \mathbb{E}\_{p(\boldsymbol{\epsilon})} \left[ f\left(g\_{\boldsymbol{\phi}}(\boldsymbol{\epsilon}, \mathbf{x})\right) \right]$, which can be **estimated** with $\frac{1}{L} \sum\_{l=1}^{L} f\left(g\_{\phi}\left(\mathbf{x}, \boldsymbol{\epsilon}^{(l)}\right)\right)$ where $\boldsymbol{\epsilon}^{(l)} \sim p(\boldsymbol{\epsilon})$. In other words, you can sample $L$ $z$ in the way we did in order to estimate $\mathbb{E}\_{\color{blue}{q\_{\boldsymbol{\phi}}(\mathbf{z} \mid \mathbf{x})}} \left[ \color{red}{f(\mathbf{z})} \right]$ (this are Monte Carlo estimates of the expectation).
  
  In the case of the VAE, we want to optimize the ELBO, which is the following objective function
  
  $$\mathcal{L}\left(\boldsymbol{\theta}, \boldsymbol{\phi} ; \mathbf{x}^{(i)}\right)=\underbrace{-D\_{K L}\left(q\_{\boldsymbol{\phi}}\left(\mathbf{z} \mid \mathbf{x}^{(i)}\right) \| p\_{\boldsymbol{\theta}}(\mathbf{z})\right)}\_{\text{KL divergence}}+ \underbrace{\mathbb{E}\_{\color{blue}{q\_{\phi}\left(\mathbf{z} \mid \mathbf{x}^{(i)}\right)}}\left[\color{red}{\log p\_{\boldsymbol{\theta}}\left(\mathbf{x}^{(i)} \mid \mathbf{z}\right)}\right]}\_{\text{likelihood}}$$
  
  which we can estimate with Monte Carlo estimates of the second term (the **likelihood term**)
  
  $$
  \widetilde{\mathcal{L}}^{B}\left(\boldsymbol{\theta}, \boldsymbol{\phi} ; \mathbf{x}^{(i)}\right)=-D\_{K L}\left(q\_{\boldsymbol{\phi}}\left(\mathbf{z} \mid \mathbf{x}^{(i)}\right) \| p\_{\boldsymbol{\theta}}(\mathbf{z})\right)+ \underbrace{\frac{1}{L} \sum\_{l=1}^{L}\left(\log p\_{\boldsymbol{\theta}}\left(\mathbf{x}^{(i)} \mid \mathbf{z}^{(i, l)}\right)\right)}\_{\text{likelihood}}
  $$
  where $$
  \text { where } \quad \mathbf{z}^{(i, l)}=g\_{\phi}\left(\boldsymbol{\epsilon}^{(i, l)}, \mathbf{x}^{(i)}\right) \text { and } \boldsymbol{\epsilon}^{(l)} \sim p(\boldsymbol{\epsilon})
  $$
  
  Here, the likelihood is computed with the neural network, for example, in practice, you use the cross-entropy of the output of your decoder, which gets as input the input to the decoder, hence $z$.
  
  You can ignore the KL divergence now because, in the case of Gaussians, [it can be computed analytically](https://ai.stackexchange.com/a/26408/2444).
  
  Now, what if we didn't use the reparametrization trick? Could we still backpropagate with respect to $\phi$? The authors of the VAE write
  
  > The usual (naïve) Monte Carlo gradient estimator for this type of problem is: $\nabla\_{\phi} \mathbb{E}\_{q\_{\phi}(\mathbf{z})}[f(\mathbf{z})]=\mathbb{E}\_{q\_{\phi}(\mathbf{z})}\left[f(\mathbf{z}) \nabla\_{q\_{\phi}(\mathbf{z})} \log q\_{\phi}(\mathbf{z})\right] \simeq \frac{1}{L} \sum\_{l=1}^{L} f(\mathbf{z}) \nabla\_{q\_{\phi}\left(\mathbf{z}^{(l)}\right)} \log q\_{\phi}\left(\mathbf{z}^{(l)}\right)$ where $\mathbf{z}^{(l)} \sim q\_{\phi}\left(\mathbf{z} \mid \mathbf{x}^{(i)}\right) .$ This gradient estimator exhibits exhibits very high variance (see e.g. [[BJP12]](https://arxiv.org/pdf/1206.6430.pdf))
  
  So, the answer is **yes**, as opposed to what many people might say around, but with another estimator (which may remind you of some equations you have seen in reinforcement learning related to [REINFORCE](http://incompleteideas.net/book/RLbook2020.pdf#page=348), if you are familiar with this), i.e. $$\frac{1}{L} \sum\_{l=1}^{L} f(\mathbf{z}) \nabla\_{q\_{\phi}\left(\mathbf{z}^{(l)}\right)} \log q\_{\phi}\left(\mathbf{z}^{(l)}\right) \tag{1}\label{1},$$ which has high variance.
  
  So, in the end, the reparametrization trick can be viewed as a [variance reduction technique](http://proceedings.mlr.press/v89/xu19a/xu19a.pdf). There are others, like [**control variates**](https://arxiv.org/pdf/1206.6430.pdf) or [**Flipout**](https://arxiv.org/abs/1803.04386) ([used e.g. in the context of Bayesian neural networks](https://github.com/tensorflow/probability/blob/main/tensorflow_probability/examples/bayesian_neural_network.py)).
  
  The first thing to note about \ref{1} is that we do not need to take the derivative with respect to $f$. The second thing is that $\nabla\_{q\_{\phi}\left(\mathbf{z}^{(l)}\right)} \log q\_{\phi}\left(\mathbf{z}^{(l)}\right)$ is the [**score function**](https://jmlr.org/papers/volume21/19-346/19-346.pdf).
  
  Now, don't ask me how to calculate this gradient $\nabla\_{q\_{\phi}\left(\mathbf{z}^{(l)}\right)} \log q\_{\phi}\left(\mathbf{z}^{(l)}\right)$ (because I am bad at math). However, note that $
  \mathbf{z}^{(l)} \sim q\_{\phi}\left(\mathbf{z} \mid \mathbf{x}^{(i)}\right)
  $, so we treat $\mathbf{z}^{(l)}$ as a fixed random sample. By the way, I am not sure whether they used $q\_{\phi}\left(\mathbf{z} \mid \mathbf{x}^{(i)}\right)$ differently from $q\_{\phi}\left(\mathbf{z} \right)$. I don't think so. I think they are the same and I think we can still back-propagate with respect to $\phi$, even if we use it to sample $\mathbf{z}^{(i)}$, because, once this latter is sampled, it can be treated as fixed (random) sample.
  
  I also note that I think they meant $\frac{1}{L} \sum\_{l=1}^{L} f(\mathbf{z}^{(l)}) \nabla\_{q\_{\phi}\left(\mathbf{z}^{(l)}\right)} \log q\_{\phi}\left(\mathbf{z}^{(l)}\right)$, i.e. they forget to use $\mathbf{z}^{(l)}$ rather than just $\mathbf{z}$ as input for $f$ in the Monte Carto estimate. You can see in section 3 of [[BJP12]](https://arxiv.org/pdf/1206.6430.pdf) and section 4.2. of [[1]](https://jmlr.org/papers/volume21/19-346/19-346.pdf) they do it like this, and it makes sense. So, the VAE paper has sloppy stuff in there.
  
  My intuition of why this has high variance is because you sample something according to the variational distribution, but then you update this variational distribution, and you continuously do this. However, I am not sure this is the right intuition. In [[BJP12]](https://arxiv.org/pdf/1206.6430.pdf), they say (in section 4) this MC estimate has high variance and that you need a lot of samples $\mathbf{x}$. I don't know exactly why this is the case because I didn't fully read this paper yet.
  
  > **Comments:**
  > **username_3**: Nice. Worth adding LHS to eq1.  
  
  ---
  
  ### Answer by username_2 on 2021-12-19. Score: 2
  
  As already mentioned in the comment, the reason, why the does the backpropagation still work is the [Reparametrization Trick](https://stats.stackexchange.com/questions/199605/how-does-the-reparameterization-trick-for-vaes-work-and-why-is-it-important).
  
  For variational autoencoder (VAE) neural networks to be learned predict parameters of the random distribution - the mean $\mu\_{\theta} (x)$ and the variance $\sigma\_{\phi} (x)$ for the case on normal distribution. Here $\theta$ and $\phi$ are parameters, that can be learned during the backprogation.
  
  At each sampling step, one generates random number $\varepsilon$ from the normal distribution with zero mean and unit variance $\mathcal{N}(0, 1)$ and then multiplies this number by $\sigma\_{\phi} (x)$ and adds $\mu\_{\theta} (x)$:
  $$
  \mu\_{\theta} (x) + \varepsilon \sigma\_{\phi} (x)
  $$
  The output of this operation has a normal distribution $\mathcal{N}(\mu\_{\theta} (x), \sigma\_{\phi} (x))$.
  You do not have to backpropagate through the epsilon, and this number is just multiplied by $\sigma\_{\phi} (x) $.
  
  The whole procedure is nicely differentiable with respect to the $\theta$ and $\phi$.
  
  However, note , that the possibility to apply this trick is allowed due to the nice property of normal distribution:
  $$
  \mathcal{N}(a, b) \sim a + b \mathcal{N}(0, 1)
  $$
  Arbitrary distribution doesn't satisfy this property.
  
  In more general case, you would need to apply some other trick to propagate the gradients.
  
  Simple strategy is the [Straight through estimator](https://arxiv.org/abs/1308.3432), when one copies the gradient from the output layer to the input through non-differentialbe operation. Or more accurate [Gumbel-Softmax softmax](https://arxiv.org/abs/1611.01144).
  
  
  ---
  
  ### Answer by username_3 on 2022-03-18. Score: 0
  
  Sampling per-se is non-differentiable in some point of view, but you can use the reparametrisation in some special scenario.
  
  Differentiation, mathematically, is the act of taking derivatives of functions wrt variables. Sampling is the act of generating a set of elements given some parameters, and differentiating a function that outputs a set is, to my knowledge, not well discussed in the setting of machine learning. More often, sampling is followed by an expectation operator, where you takes average of a function across your samples. And this expectation operator, is differentiable.
  
  $$E\_\pi(X(s))={1\over N} \sum\_{i=1,\{s\_i\}\sim\pi}^N X(s\_i)=\sum\_s X(s)P\_\pi(s)$$
  
  In fact, backpropagating through a sampling operation has a well known name -- it's called "policy gradient" in reinforcement learning, where you study how to take samples (known as policy) so as to minimise error (known as maximising the expected reward).
  
  Not sure why this connection has not been well studied yet... maybe NN people just love a fully deterministic function approximator.
  
  > **Comments:**
  > **username_4**: You don't have to leave the "notice" in your new answer. Moderators can convert answer-like comments to comments or even delete them. This is not weird here because we expect users to provide answers with details and some users might have written a comment as an answer because they don't know how the site works or maybe they don't have the reputation to comment. Having said that, you can post another answer, as you did, once you have written a more detailed one.  
  > **username_3**: I just wanted to say, that I was not happy, and the moderator is now removing my notice. My emotion was not acknowledged -- it was REMOVED. I don't feel my voice respected anymore. People are not machines.  
  > **username_4**: I removed the noticed, and I am currently not a mod. So it was not a moderator that removed the notice. People that have more than 2k rep here can edit posts without confirmation, like I did. Our site's model discourages people to leave unnecessary details in the questions or answers, with respect to the question being asked. Things like "Thanks!" or "Have a nice day" are discouraged in questions or answers. It's not because we're nice, but because we want to create an archive of questions and answers where people can easily find their answer without losing time reading unnecessary details.  
  > **username_4**: I understand your feelings, but you have to understand that this site is not meant for people to share their opinions, but to give answers to questions. Anyway, thanks for trying to contributing to site. If you haven't done it yet, you might want to have a look at our site [on-topic page](https://ai.stackexchange.com/help/on-topic).  
  > **username_3**: Ok thanks. I am happy to see we are actually trying to communicate. I agree with the general intention that to keep the site well-structured, but editing is a strong privilege and the mechanism could definitely be improved to balance between motivating contribution while keep the quality high.  
  
  ---
  
  