Answers.Body: <p>The <a href="https://arxiv.org/pdf/1312.6114.pdf" rel="nofollow noreferrer">VAE</a> attempts to model a specific probabilistic (directed) graphical model (Bayesian network)</p>
  <p><a href="https://i.sstatic.net/ZE52W.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ZE52W.png" alt="enter image description here" /></a></p>
  <p>So, in this PGM, <span class="math-container">$\mathbf{z}$</span> and <span class="math-container">$\mathbf{x}$</span> are random variables. In principle, I think you could also model <span class="math-container">$\phi$</span> and <span class="math-container">$\theta$</span> as random variables (in Bayesian statistics, you can also model parameters as random variables and put priors on them).</p>
  <p>In practice, the VAE attempts to learn a generative model given a dataset. In fact, the technical part of the VAE paper (section 2.1) starts with</p>
  <blockquote>
  <p>Let us consider some <strong>dataset</strong> <span class="math-container">$\mathbf{X}=\left\{\mathbf{x}^{(i)}\right\}_{i=1}^{N}$</span> consisting of <span class="math-container">$N$</span> i.i.d. samples of some continuous or discrete variable <span class="math-container">$\mathbf{x}$</span>. We assume that the data are generated by some random process, involving an unobserved continuous random variable <span class="math-container">$\mathbf{z}$</span>.</p>
  </blockquote>
  <p>Later, they use this dataset to define the <strong>likelihood</strong></p>
  <p><span class="math-container">$$\log p_{\theta}\left(\mathbf{x}^{(i)}\right)=D_{K L}\left(q_{\phi}\left(\mathbf{z} \mid \mathbf{x}^{(i)}\right) \| p_{\theta}\left(\mathbf{z} \mid \mathbf{x}^{(i)}\right)\right)+\mathcal{L}\left(\boldsymbol{\theta}, \boldsymbol{\phi} ; \mathbf{x}^{(i)}\right)$$</span></p>
  <p>and, consequently, also the objective function (the Evidence Lower BOund, aka ELBO).</p>
  <p><span class="math-container">$$
  \mathcal{L}\left(\boldsymbol{\theta}, \phi ; \mathbf{x}^{(i)}\right)=-D_{K L}\left(q_{\phi}\left(\mathbf{z} \mid \mathbf{x}^{(i)}\right) \| p_{\theta}(\mathbf{z})\right)+\mathbb{E}_{q_{\phi}\left(\mathbf{z} \mid \mathbf{x}^{(i)}\right)}\left[\log p_{\theta}\left(\mathbf{x}^{(i)} \mid \mathbf{z}\right)\right]
  $$</span></p>
  <p>Note that we use <span class="math-container">$\mathbf{x}^{(i)}$</span> in the formulas above, so the distributions are conditioned on the given samples/dataset and the likelihood function is defined in terms of the samples (that's what a likelihood usually is: a function of the parameters given the usually fixed data).</p>
  <p>In the other <a href="https://arxiv.org/pdf/1906.02691.pdf" rel="nofollow noreferrer">more extensive paper about VAEs</a> (by the same authors of the VAE), you also have a diagram of the VAE.</p>
  <p><a href="https://i.sstatic.net/70YvW.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/70YvW.png" alt="enter image description here" /></a></p>
  <p>So, I think an answer to your question depends on what you actually want to show with the diagram. If you want to show how VAEs are trained, then you definitely need to show that we have a dataset. If your diagram is supposed to show the distributions that the VAE attempts to model, then you can probably use a PGM.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-17T02:35:59.760Z
Answers.Text: So like when we are sampling from the encoder, we use the reparameterization trick, $\epsilon \sim p(\epsilon)$, $z=\mu+ z\odot \sigma$, and then input $z$ into the decoder. As far as I know, when we are sampling from a distribution, like $\epsilon \sim p(\epsilon)$, $\epsilon$ is a random variable/vector, and then . Is there some point along the way that we start treating it (either $\epsilon$ or $z$) as an instance instead of a random variable/vector?
Answers.UserDisplayName: a12345
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-17T08:59:13.783Z
Answers.Text: @a12345 Before you sample, you can think that there's a random variable $\epsilon$ that follows a distribution $p$. Once you sample $z$ using the re-parametrization trick, you can think of it as a real data sample/instance. I think that people talk about random variables a little bit imprecisely in machine learning. Sometimes, they might say that $\epsilon$ is a random variable because it was randomly sampled from a distribution. What we can say is that, once we sample $z$, which we pass to the decoder, $z$ is no more a random variable, it's a (random) sample, i.a. sample chosen randomly.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-17T09:04:33.437Z
Answers.Text: Note that, in my comment above, I used the letter $\epsilon$ once to refer to a random variable and other times to refer to a sample from that probability distribution associated with that random variable.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-15T10:00:38.020Z
Answers.LastActivityDate: 2021-12-15T10:00:38.020Z
Answers.UserDisplayName: nbro
Body: <p>For a variational autoencoder, we have input <span class="math-container">$x$</span> (assume 1 data point for now, like an image), a latent code sampled from the decoder, <span class="math-container">$z$</span>, and an output <span class="math-container">$\hat{x}$</span>.</p>
  <p>If I were to draw a diagram for the VAE with the input, output, and latent code sample, is it appropriate to write those three as random variables/vectors? Or as instances of random variables/vectors?</p>
  <p>I thought it was random variables/vectors, but I saw <a href="https://stats.stackexchange.com/questions/358342/what-the-relation-between-a-random-variable-and-a-sample-or-dataset-in-machine?rq=1">this discussion</a>, where they talk about the dataset being instances.</p>
  
Tags: <neural-networks><variational-autoencoder><random-variable><probabilistic-graphical-models>
Title: For the VAE, should the input, output and latent variable code be random variables?
UserDisplayName: a12345
ThreadText: # For the VAE, should the input, output and latent variable code be random variables?
  
  (Asked by: username_0 on 2021-12-15)
  
  For a variational autoencoder, we have input $x$ (assume 1 data point for now, like an image), a latent code sampled from the decoder, $z$, and an output $\hat{x}$.
  
  If I were to draw a diagram for the VAE with the input, output, and latent code sample, is it appropriate to write those three as random variables/vectors? Or as instances of random variables/vectors?
  
  I thought it was random variables/vectors, but I saw [this discussion](https://stats.stackexchange.com/questions/358342/what-the-relation-between-a-random-variable-and-a-sample-or-dataset-in-machine?rq=1), where they talk about the dataset being instances.
  
  **Tags:** neural-networks, variational-autoencoder, random-variable, probabilistic-graphical-models
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-12-15. Score: 1 | Accepted answer
  
  The [VAE](https://arxiv.org/pdf/1312.6114.pdf) attempts to model a specific probabilistic (directed) graphical model (Bayesian network)
  
  [![enter image description here](https://i.sstatic.net/ZE52W.png)](https://i.sstatic.net/ZE52W.png)
  
  So, in this PGM, $\mathbf{z}$ and $\mathbf{x}$ are random variables. In principle, I think you could also model $\phi$ and $\theta$ as random variables (in Bayesian statistics, you can also model parameters as random variables and put priors on them).
  
  In practice, the VAE attempts to learn a generative model given a dataset. In fact, the technical part of the VAE paper (section 2.1) starts with
  
  > Let us consider some **dataset** $\mathbf{X}=\left\{\mathbf{x}^{(i)}\right\}\_{i=1}^{N}$ consisting of $N$ i.i.d. samples of some continuous or discrete variable $\mathbf{x}$. We assume that the data are generated by some random process, involving an unobserved continuous random variable $\mathbf{z}$.
  
  Later, they use this dataset to define the **likelihood**
  
  $$\log p\_{\theta}\left(\mathbf{x}^{(i)}\right)=D\_{K L}\left(q\_{\phi}\left(\mathbf{z} \mid \mathbf{x}^{(i)}\right) \| p\_{\theta}\left(\mathbf{z} \mid \mathbf{x}^{(i)}\right)\right)+\mathcal{L}\left(\boldsymbol{\theta}, \boldsymbol{\phi} ; \mathbf{x}^{(i)}\right)$$
  
  and, consequently, also the objective function (the Evidence Lower BOund, aka ELBO).
  
  $$
  \mathcal{L}\left(\boldsymbol{\theta}, \phi ; \mathbf{x}^{(i)}\right)=-D\_{K L}\left(q\_{\phi}\left(\mathbf{z} \mid \mathbf{x}^{(i)}\right) \| p\_{\theta}(\mathbf{z})\right)+\mathbb{E}\_{q\_{\phi}\left(\mathbf{z} \mid \mathbf{x}^{(i)}\right)}\left[\log p\_{\theta}\left(\mathbf{x}^{(i)} \mid \mathbf{z}\right)\right]
  $$
  
  Note that we use $\mathbf{x}^{(i)}$ in the formulas above, so the distributions are conditioned on the given samples/dataset and the likelihood function is defined in terms of the samples (that's what a likelihood usually is: a function of the parameters given the usually fixed data).
  
  In the other [more extensive paper about VAEs](https://arxiv.org/pdf/1906.02691.pdf) (by the same authors of the VAE), you also have a diagram of the VAE.
  
  [![enter image description here](https://i.sstatic.net/70YvW.png)](https://i.sstatic.net/70YvW.png)
  
  So, I think an answer to your question depends on what you actually want to show with the diagram. If you want to show how VAEs are trained, then you definitely need to show that we have a dataset. If your diagram is supposed to show the distributions that the VAE attempts to model, then you can probably use a PGM.
  
  > **Comments:**
  > **username_0**: So like when we are sampling from the encoder, we use the reparameterization trick, $\epsilon \sim p(\epsilon)$, $z=\mu+ z\odot \sigma$, and then input $z$ into the decoder. As far as I know, when we are sampling from a distribution, like $\epsilon \sim p(\epsilon)$, $\epsilon$ is a random variable/vector, and then . Is there some point along the way that we start treating it (either $\epsilon$ or $z$) as an instance instead of a random variable/vector?  
  > **username_1**: @username_0 Before you sample, you can think that there's a random variable $\epsilon$ that follows a distribution $p$. Once you sample $z$ using the re-parametrization trick, you can think of it as a real data sample/instance. I think that people talk about random variables a little bit imprecisely in machine learning. Sometimes, they might say that $\epsilon$ is a random variable because it was randomly sampled from a distribution. What we can say is that, once we sample $z$, which we pass to the decoder, $z$ is no more a random variable, it's a (random) sample, i.a. sample chosen randomly.  
  > **username_1**: Note that, in my comment above, I used the letter $\epsilon$ once to refer to a random variable and other times to refer to a sample from that probability distribution associated with that random variable.  
  
  ---
  
  