Answers.Body: <p>Yes, in the case of the Gaussian, you have two distinct layers (so weights and biases), one for the mean and the other for the variance, as the equations are telling us.</p>
  <p>The mean is calculated with the weights <span class="math-container">$\mathbf{W}_{4}$</span> and bias <span class="math-container">$\mathbf{b}_{4}$</span> from <span class="math-container">$\mathbf{h}$</span> as follows</p>
  <p><span class="math-container">$$\boldsymbol{\mu} =\mathbf{W}_{4} \mathbf{h}+\mathbf{b}_{4},$$</span></p>
  <p>while the variance (actually, equivalently, the log of the standard deviation) is calculated from <span class="math-container">$\mathbf{W}_{5}$</span> and <span class="math-container">$\mathbf{b}_{5}$</span> from <span class="math-container">$\mathbf{h}$</span> as follows</p>
  <p><span class="math-container">$$\log \sigma^{2} =\mathbf{W}_{5} \mathbf{h}+\mathbf{b}_{5}$$</span></p>
  <p><a href="https://github.com/pytorch/examples/blob/151944ecaf9ba2c8288ee550143ae7ffdaa90a80/vae/main.py#L49" rel="nofollow noreferrer">Here</a> you have a PyTorch implementation that uses 2 distinct linear/dense layers for doing this, but note that it is doing this only for the encoder to produce the latent vector <span class="math-container">$\mathbf{z}$</span>.</p>
  <p>Yes, generally, these layers (so the mean and variance) are not the same for the encoder and decoder. However, it would not be surprising to me if someone already tried to share some layers between the encoder and decoder for some specific task.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-05T09:05:10.133Z
Answers.LastActivityDate: 2021-12-05T09:05:10.133Z
Answers.UserDisplayName: nbro
Body: <p>I am looking at appendix C of <a href="https://arxiv.org/pdf/1312.6114.pdf" rel="nofollow noreferrer">the VAE paper</a>:</p>
  <p>It says:</p>
  <blockquote>
  <p><strong>C.1 Bernoulli MLP as decoder</strong></p>
  <p>In this case let <span class="math-container">$p_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z})$</span> be a multivariate Bernoulli whose probabilities are computed from <span class="math-container">$\mathrm{z}$</span> with a fully-connected neural network with a single hidden layer:
  <span class="math-container">$$
  \begin{aligned}
  \log p(\mathbf{x} \mid \mathbf{z}) &amp;=\sum_{i=1}^{D} x_{i} \log y_{i}+\left(1-x_{i}\right) \cdot \log \left(1-y_{i}\right) \\
  \text { where } \mathbf{y} &amp;=f_{\sigma}\left(\mathbf{W}_{2} \tanh \left(\mathbf{W}_{1} \mathbf{z}+\mathbf{b}_{1}\right)+\mathbf{b}_{2}\right)
  \end{aligned}
  $$</span>
  where <span class="math-container">$f_{\sigma}(.)$</span> is the elementwise sigmoid activation function, and where <span class="math-container">$\theta=\left\{\mathbf{W}_{1}, \mathbf{W}_{2}, \mathbf{b}_{1}, \mathbf{b}_{2}\right\}$</span> are the weights and biases of the MLP.</p>
  <p><strong>C.2 Gaussian MLP as encoder or decoder</strong></p>
  <p>In this case let encoder or decoder be a multivariate Gaussian with a diagonal covariance structure:
  <span class="math-container">$$
  \begin{aligned}
  \log p(\mathbf{x} \mid \mathbf{z}) &amp;=\log \mathcal{N}\left(\mathbf{x} ; \boldsymbol{\mu}, \boldsymbol{\sigma}^{2} \mathbf{I}\right) \\
  \text { where } \boldsymbol{\mu} &amp;=\mathbf{W}_{4} \mathbf{h}+\mathbf{b}_{4} \\
  \log \sigma^{2} &amp;=\mathbf{W}_{5} \mathbf{h}+\mathbf{b}_{5} \\
  \mathbf{h} &amp;=\tanh \left(\mathbf{W}_{3} \mathbf{z}+\mathbf{b}_{3}\right)
  \end{aligned}
  $$</span>
  where <span class="math-container">$\left\{\mathbf{W}_{3}, \mathbf{W}_{4}, \mathbf{W}_{5}, \mathbf{b}_{3}, \mathbf{b}_{4}, \mathbf{b}_{5}\right\}$</span> are the weights and biases of the MLP and part of <span class="math-container">$\boldsymbol{\theta}$</span> when used as decoder. Note that when this network is used as an encoder <span class="math-container">$q_{\phi}(\mathbf{z} \mid \mathbf{x})$</span>, then <span class="math-container">$\mathrm{z}$</span> and <span class="math-container">$\mathrm{x}$</span> are swapped, and the weights and biases are variational parameters <span class="math-container">$\phi$</span>.</p>
  </blockquote>
  <p>So, it seems like, for a Bernoulli decoder, it only outputs a vector <span class="math-container">$\mathbf{y}$</span>, which gets plugged into the log-likelihood formula.
  But then, for the Gaussian decoder, it outputs both <span class="math-container">$\boldsymbol{\sigma}$</span> and <span class="math-container">$\mu$</span>. So, is it like 2 parallel layers, one calculating  <span class="math-container">$\boldsymbol{\sigma}$</span>  one calculating <span class="math-container">$\mu$</span>?</p>
  <p>Similar to how we get the <span class="math-container">$\mu$</span> and <span class="math-container">$\sigma$</span> of the encoder (which I am assuming the encoder ones are different from the decoder ones)?</p>
  <p>And we plug it into the formula I derived in this link
  <a href="https://stats.stackexchange.com/questions/540092/how-do-we-get-to-the-mse-in-the-loss-function-for-a-variational-autoencoder">here</a>,  the log-likelihood to get the reconstruction loss?</p>
  <p>This is the intuition I am getting, but I haven't seen it explicitly all in one place.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-08T13:29:05.723Z
Comments.Text: [You should accept my answer if answers your question](https://ai.stackexchange.com/help/someone-answers). If it doesn't answer your question, can you explain why? Note also that, ideally, each post should contain only one question, so, please, next time, ask one question per post. If you have multiple questions, ask each of them in a separate post (but make sure to provide the necessary context to understand each question in their separate post).
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-08T13:44:22.440Z
Comments.Text: In particular, your last question is quite distinct from the other ones, so, please, ask it in a separate post. It seems that that last question is: "_What the loss should be when the decoder is also a Gaussian rather than a Bernoulli_?" This question may pop multiple times, so I think it deserves its own post with a good answer.
Comments.UserDisplayName: nbro
Tags: <deep-learning><papers><variational-autoencoder><variational-inference>
Title: Do we use two distinct layers to compute the mean and variance of a Gaussian encoder/decoder in the VAE?
UserDisplayName: a12345
ThreadText: # Do we use two distinct layers to compute the mean and variance of a Gaussian encoder/decoder in the VAE?
  
  (Asked by: username_0 on 2021-12-03)
  
  I am looking at appendix C of [the VAE paper](https://arxiv.org/pdf/1312.6114.pdf):
  
  It says:
  
  > **C.1 Bernoulli MLP as decoder**
  >
  > In this case let $p\_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z})$ be a multivariate Bernoulli whose probabilities are computed from $\mathrm{z}$ with a fully-connected neural network with a single hidden layer:
  > $$
  > \begin{aligned}
  > \log p(\mathbf{x} \mid \mathbf{z}) &=\sum\_{i=1}^{D} x\_{i} \log y\_{i}+\left(1-x\_{i}\right) \cdot \log \left(1-y\_{i}\right) \\
  > \text { where } \mathbf{y} &=f\_{\sigma}\left(\mathbf{W}\_{2} \tanh \left(\mathbf{W}\_{1} \mathbf{z}+\mathbf{b}\_{1}\right)+\mathbf{b}\_{2}\right)
  > \end{aligned}
  > $$
  > where $f\_{\sigma}(.)$ is the elementwise sigmoid activation function, and where $\theta=\left\{\mathbf{W}\_{1}, \mathbf{W}\_{2}, \mathbf{b}\_{1}, \mathbf{b}\_{2}\right\}$ are the weights and biases of the MLP.
  >
  > **C.2 Gaussian MLP as encoder or decoder**
  >
  > In this case let encoder or decoder be a multivariate Gaussian with a diagonal covariance structure:
  > $$
  > \begin{aligned}
  > \log p(\mathbf{x} \mid \mathbf{z}) &=\log \mathcal{N}\left(\mathbf{x} ; \boldsymbol{\mu}, \boldsymbol{\sigma}^{2} \mathbf{I}\right) \\
  > \text { where } \boldsymbol{\mu} &=\mathbf{W}\_{4} \mathbf{h}+\mathbf{b}\_{4} \\
  > \log \sigma^{2} &=\mathbf{W}\_{5} \mathbf{h}+\mathbf{b}\_{5} \\
  > \mathbf{h} &=\tanh \left(\mathbf{W}\_{3} \mathbf{z}+\mathbf{b}\_{3}\right)
  > \end{aligned}
  > $$
  > where $\left\{\mathbf{W}\_{3}, \mathbf{W}\_{4}, \mathbf{W}\_{5}, \mathbf{b}\_{3}, \mathbf{b}\_{4}, \mathbf{b}\_{5}\right\}$ are the weights and biases of the MLP and part of $\boldsymbol{\theta}$ when used as decoder. Note that when this network is used as an encoder $q\_{\phi}(\mathbf{z} \mid \mathbf{x})$, then $\mathrm{z}$ and $\mathrm{x}$ are swapped, and the weights and biases are variational parameters $\phi$.
  
  So, it seems like, for a Bernoulli decoder, it only outputs a vector $\mathbf{y}$, which gets plugged into the log-likelihood formula.
  But then, for the Gaussian decoder, it outputs both $\boldsymbol{\sigma}$ and $\mu$. So, is it like 2 parallel layers, one calculating $\boldsymbol{\sigma}$ one calculating $\mu$?
  
  Similar to how we get the $\mu$ and $\sigma$ of the encoder (which I am assuming the encoder ones are different from the decoder ones)?
  
  And we plug it into the formula I derived in this link
  [here](https://stats.stackexchange.com/questions/540092/how-do-we-get-to-the-mse-in-the-loss-function-for-a-variational-autoencoder), the log-likelihood to get the reconstruction loss?
  
  This is the intuition I am getting, but I haven't seen it explicitly all in one place.
  
  **Tags:** deep-learning, papers, variational-autoencoder, variational-inference
  
  > **Comments:**
  > **username_1**: [You should accept my answer if answers your question](https://ai.stackexchange.com/help/someone-answers). If it doesn't answer your question, can you explain why? Note also that, ideally, each post should contain only one question, so, please, next time, ask one question per post. If you have multiple questions, ask each of them in a separate post (but make sure to provide the necessary context to understand each question in their separate post).  
  > **username_1**: In particular, your last question is quite distinct from the other ones, so, please, ask it in a separate post. It seems that that last question is: "_What the loss should be when the decoder is also a Gaussian rather than a Bernoulli_?" This question may pop multiple times, so I think it deserves its own post with a good answer.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-12-05. Score: 1 | Accepted answer
  
  Yes, in the case of the Gaussian, you have two distinct layers (so weights and biases), one for the mean and the other for the variance, as the equations are telling us.
  
  The mean is calculated with the weights $\mathbf{W}\_{4}$ and bias $\mathbf{b}\_{4}$ from $\mathbf{h}$ as follows
  
  $$\boldsymbol{\mu} =\mathbf{W}\_{4} \mathbf{h}+\mathbf{b}\_{4},$$
  
  while the variance (actually, equivalently, the log of the standard deviation) is calculated from $\mathbf{W}\_{5}$ and $\mathbf{b}\_{5}$ from $\mathbf{h}$ as follows
  
  $$\log \sigma^{2} =\mathbf{W}\_{5} \mathbf{h}+\mathbf{b}\_{5}$$
  
  [Here](https://github.com/pytorch/examples/blob/151944ecaf9ba2c8288ee550143ae7ffdaa90a80/vae/main.py#L49) you have a PyTorch implementation that uses 2 distinct linear/dense layers for doing this, but note that it is doing this only for the encoder to produce the latent vector $\mathbf{z}$.
  
  Yes, generally, these layers (so the mean and variance) are not the same for the encoder and decoder. However, it would not be surprising to me if someone already tried to share some layers between the encoder and decoder for some specific task.
  
  
  ---
  
  