Answers.Body: <p>I think this is very confusing to many people. I had to deal with VAEs (and Bayesian neural networks) multiple times in the past, and I've seen so many inconsistent notations and unclear explanations. Here's what I know about the issue that you're raising, but I will not give you a definitive answer.</p>
  <p>In the appendix of <a href="https://arxiv.org/pdf/1312.6114.pdf" rel="nofollow noreferrer">the VAE paper</a> (sections C.1 and C.2), they consider the cases of</p>
  <ol>
  <li>a multivariate Bernoulli decoder, and</li>
  <li>Gaussian encoder and decoder</li>
  </ol>
  <p>They write</p>
  <ul>
  <li><p>in the case of the multivariate Bernoulli decoder, <span class="math-container">$\theta$</span> are the weights and biases of the neural network that outputs the Bernoulli parameters (each Bernoulli distribution has only one parameter that basically defines the probability of getting one sample rather than the other - this is not written in that section - I am just telling you this, in case you didn't know it).</p>
  </li>
  <li><p>in the case of the Gaussian encoder, <span class="math-container">$\phi$</span> (which parametrizes the variational distribution <span class="math-container">$q_\phi$</span>) are the weights and biases of the neural network that represents the encoder (which outputs the <span class="math-container">$\mu$</span> and <span class="math-container">$\sigma$</span> of the Gaussian, which are the parameters that define the specific Gaussian - you can see this from the equations in appendix C.2)</p>
  </li>
  <li><p>in the case of the Gaussian decoder (appendix C.2), the weights and biases of the neural network (the decoder) are <strong>part of</strong> <span class="math-container">$\theta$</span> (I don't know what they mean by &quot;part of&quot; here, honestly).</p>
  </li>
  </ul>
  <p>So, in the case of the VAE (with a multivariate Bernoulli decoder, or a Gaussian encoder and decoder), which is really a neural network (or 2 neural networks - encoder and decoder) that attempts to model the random process which is represented by the graphical model in figure 1 of <a href="https://arxiv.org/pdf/1312.6114.pdf" rel="nofollow noreferrer">the VAE paper</a>, <span class="math-container">$\phi$</span> and <span class="math-container">$\theta$</span> are the weights and biases of the neural networks that represent the encoder and decoder (partially, in the case of the Gaussian decoder, they say), respectively. <span class="math-container">$\phi$</span> and <span class="math-container">$\theta$</span> are really the learnable parameters. In fact, if you look at algorithm 1 or any implementation of the VAE (I like <a href="https://github.com/pytorch/examples/blob/main/vae/main.py" rel="nofollow noreferrer">this simple PyTorch one</a>), they are what we're trying to optimize.</p>
  <p>Now, if you look at the definition of a parametric model, e.g. <a href="https://en.wikipedia.org/wiki/Parametric_model" rel="nofollow noreferrer">here</a>, you will see that the parameters <span class="math-container">$\theta$</span> in <span class="math-container">$p_\theta$</span> actually refer to e.g. the mean and variance of the Gaussian, i.e. <span class="math-container">$\theta = \{ \mu, \sigma \}$</span>. This is one thing that causes confusion - <strong>the subscript <span class="math-container">$\theta$</span> (or <span class="math-container">$\phi$</span>) refers to different things in different articles.</strong></p>
  <p>HOWEVER, ultimately, what we're trying to learn are the parameters of a probability distribution, let's say it's a Gaussian, so we are trying to learn a mean and variance <span class="math-container">$\{\mu, \sigma \}$</span>. In the VAE, we use a neural network to output the mean and variance, i.e. <span class="math-container">$\mu, \sigma = f_\phi(x)$</span>, where <span class="math-container">$f_\phi$</span> is the neural network with weights <span class="math-container">$\theta$</span> (see the equations in appendix C) and <span class="math-container">$x$</span> is an input. So, <span class="math-container">$\{\mu, \sigma \}$</span> are defined by <span class="math-container">$\phi$</span>.</p>
  <p>Now, back to your actual question. Here are some possible interpretations of that notation.</p>
  <ol>
  <li><p>If <span class="math-container">$\theta$</span> represented the parameters of a specific probability distribution (let's say a Gaussian), so <span class="math-container">$\theta = \{ \mu, \sigma \}$</span>, then, as you noticed, you would have all your distributions with the same mean and variance, and I don't think this is generally the case (but could happen?) - for example, <span class="math-container">$p_\theta(x)$</span> and <span class="math-container">$p_\theta(z)$</span> would exactly be the same distribution, so that means that <span class="math-container">$x$</span> and <span class="math-container">$z$</span> follow exactly the same distribution (e.g. <span class="math-container">$\mathcal{N}(0, 1)$</span>).</p>
  </li>
  <li><p>However, if you look at figure 1 of <a href="https://arxiv.org/pdf/1312.6114.pdf" rel="nofollow noreferrer">the VAE paper</a>, both <span class="math-container">$x$</span> and <span class="math-container">$z$</span> depend on <span class="math-container">$\theta$</span>, but is it the same <span class="math-container">$\theta$</span>? So, even though the letter <span class="math-container">$\theta$</span> is used as a subscript in the likelihood, prior, marginal and posterior, do these distributions actually have the same specific values of <span class="math-container">$\theta$</span>? If you look at section 2.1, they also use that notation you found on Wikipedia.</p>
  </li>
  <li><p>If <span class="math-container">$\theta$</span> represented the weights of neural networks, then all neural networks would be equal (this is not the case in practice - see <a href="https://github.com/pytorch/examples/blob/main/vae/main.py" rel="nofollow noreferrer">the PyTorch implementation</a>).</p>
  </li>
  <li><p>If <span class="math-container">$\theta$</span> were <a href="https://www.statisticshowto.com/hyperparameter/" rel="nofollow noreferrer">hyper-parameters</a>, that notation could make sense (because the mean and variance of the Gaussians could still be different), but this does not seem to be consistent with the actual explanation in appendix C and the PyTorch implementation.</p>
  </li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-02T10:53:22.177Z
Answers.Text: In most cases θ is used to indicate weights of a neural network, and with sloppy notation that could mean any NN so not necessarily the same one every time.
Answers.UserDisplayName: N. Kiefer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-04T11:45:47.393Z
Answers.LastActivityDate: 2022-06-05T08:37:12.413Z
Answers.LastEditDate: 2022-06-05T08:37:12.413Z
Answers.UserDisplayName: nbro
Body: <p>I've noticed in several resources on variational autoencoders (for example the <a href="https://en.wikipedia.org/wiki/Variational_autoencoder" rel="nofollow noreferrer">wikipedia article</a>), we use the same parameters theta for the prior, likelihood, posterior, etc distributions. For example the equation <span class="math-container">$p_\theta(x) = \int_z p_\theta(x|z)p_\theta(z)dz$</span>. Aren't <span class="math-container">$p_\theta(x)$</span> and  <span class="math-container">$p_\theta(z)$</span> two different distributions, so how can we parameterize them with the same <span class="math-container">$\theta$</span> params. I might be misunderstanding something about what it means to parameterize a distribution...</p>
  
Tags: <machine-learning><generative-adversarial-networks><generative-model><variational-autoencoder>
Title: Why do we use the same parameters for the joint, marginal and conditional distributions in VAEs?
UserDisplayName: Marko
ThreadText: # Why do we use the same parameters for the joint, marginal and conditional distributions in VAEs?
  
  (Asked by: username_0 on 2022-05-28)
  
  I've noticed in several resources on variational autoencoders (for example the [wikipedia article](https://en.wikipedia.org/wiki/Variational_autoencoder)), we use the same parameters theta for the prior, likelihood, posterior, etc distributions. For example the equation $p\_\theta(x) = \int\_z p\_\theta(x|z)p\_\theta(z)dz$. Aren't $p\_\theta(x)$ and $p\_\theta(z)$ two different distributions, so how can we parameterize them with the same $\theta$ params. I might be misunderstanding something about what it means to parameterize a distribution...
  
  **Tags:** machine-learning, generative-adversarial-networks, generative-model, variational-autoencoder
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-06-04. Score: 1
  
  I think this is very confusing to many people. I had to deal with VAEs (and Bayesian neural networks) multiple times in the past, and I've seen so many inconsistent notations and unclear explanations. Here's what I know about the issue that you're raising, but I will not give you a definitive answer.
  
  In the appendix of [the VAE paper](https://arxiv.org/pdf/1312.6114.pdf) (sections C.1 and C.2), they consider the cases of
  
  1. a multivariate Bernoulli decoder, and
  2. Gaussian encoder and decoder
  
  They write
  
  * in the case of the multivariate Bernoulli decoder, $\theta$ are the weights and biases of the neural network that outputs the Bernoulli parameters (each Bernoulli distribution has only one parameter that basically defines the probability of getting one sample rather than the other - this is not written in that section - I am just telling you this, in case you didn't know it).
  * in the case of the Gaussian encoder, $\phi$ (which parametrizes the variational distribution $q\_\phi$) are the weights and biases of the neural network that represents the encoder (which outputs the $\mu$ and $\sigma$ of the Gaussian, which are the parameters that define the specific Gaussian - you can see this from the equations in appendix C.2)
  * in the case of the Gaussian decoder (appendix C.2), the weights and biases of the neural network (the decoder) are **part of** $\theta$ (I don't know what they mean by "part of" here, honestly).
  
  So, in the case of the VAE (with a multivariate Bernoulli decoder, or a Gaussian encoder and decoder), which is really a neural network (or 2 neural networks - encoder and decoder) that attempts to model the random process which is represented by the graphical model in figure 1 of [the VAE paper](https://arxiv.org/pdf/1312.6114.pdf), $\phi$ and $\theta$ are the weights and biases of the neural networks that represent the encoder and decoder (partially, in the case of the Gaussian decoder, they say), respectively. $\phi$ and $\theta$ are really the learnable parameters. In fact, if you look at algorithm 1 or any implementation of the VAE (I like [this simple PyTorch one](https://github.com/pytorch/examples/blob/main/vae/main.py)), they are what we're trying to optimize.
  
  Now, if you look at the definition of a parametric model, e.g. [here](https://en.wikipedia.org/wiki/Parametric_model), you will see that the parameters $\theta$ in $p\_\theta$ actually refer to e.g. the mean and variance of the Gaussian, i.e. $\theta = \{ \mu, \sigma \}$. This is one thing that causes confusion - **the subscript $\theta$ (or $\phi$) refers to different things in different articles.**
  
  HOWEVER, ultimately, what we're trying to learn are the parameters of a probability distribution, let's say it's a Gaussian, so we are trying to learn a mean and variance $\{\mu, \sigma \}$. In the VAE, we use a neural network to output the mean and variance, i.e. $\mu, \sigma = f\_\phi(x)$, where $f\_\phi$ is the neural network with weights $\theta$ (see the equations in appendix C) and $x$ is an input. So, $\{\mu, \sigma \}$ are defined by $\phi$.
  
  Now, back to your actual question. Here are some possible interpretations of that notation.
  
  1. If $\theta$ represented the parameters of a specific probability distribution (let's say a Gaussian), so $\theta = \{ \mu, \sigma \}$, then, as you noticed, you would have all your distributions with the same mean and variance, and I don't think this is generally the case (but could happen?) - for example, $p\_\theta(x)$ and $p\_\theta(z)$ would exactly be the same distribution, so that means that $x$ and $z$ follow exactly the same distribution (e.g. $\mathcal{N}(0, 1)$).
  2. However, if you look at figure 1 of [the VAE paper](https://arxiv.org/pdf/1312.6114.pdf), both $x$ and $z$ depend on $\theta$, but is it the same $\theta$? So, even though the letter $\theta$ is used as a subscript in the likelihood, prior, marginal and posterior, do these distributions actually have the same specific values of $\theta$? If you look at section 2.1, they also use that notation you found on Wikipedia.
  3. If $\theta$ represented the weights of neural networks, then all neural networks would be equal (this is not the case in practice - see [the PyTorch implementation](https://github.com/pytorch/examples/blob/main/vae/main.py)).
  4. If $\theta$ were [hyper-parameters](https://www.statisticshowto.com/hyperparameter/), that notation could make sense (because the mean and variance of the Gaussians could still be different), but this does not seem to be consistent with the actual explanation in appendix C and the PyTorch implementation.
  
  > **Comments:**
  > **username_2**: In most cases θ is used to indicate weights of a neural network, and with sloppy notation that could mean any NN so not necessarily the same one every time.  
  
  ---
  
  