Body: <p>Following the standard setup/notation for a VAE, let <span class="math-container">$z$</span> denote the latent variables, <span class="math-container">$q$</span> as the encoder, <span class="math-container">$p$</span> as the decoder, and <span class="math-container">$x$</span> as the label. Let the objective be to maximize the ELBO, where a single sample monte carlo estimate of the ELBO is given by
  <span class="math-container">\begin{align*} 
  \log p(x \, | \, z) + \log p(z) - \log q(z \, | \, x)
  \end{align*}</span></p>
  <p>Now I want to focus only on the <span class="math-container">$\log p(x \, | \, z)$</span> term, where <span class="math-container">$x$</span> is an image, and the decoder <span class="math-container">$p$</span> outputs the mean/variance of a normal distribution for each pixel.</p>
  <p>My understanding is that pixel values should be integer 0-255. Now consider a single pixel: suppose that the ground truth for that pixel is the value 10, and the encoder predicts the mean, variance <span class="math-container">$\mu, \sigma^2$</span> respectively. Now when computing the ELBO, we have this term
  <span class="math-container">\begin{align*} 
  \log p(x \, | \, z) = \log f(10, \mu, \sigma^2) \tag{1}
  \end{align*}</span>
  where <span class="math-container">$f$</span> is the probability density of the normal distribution. My question is why it is justified to compute <span class="math-container">$\log p(x \, | \, z)$</span> using the density considering that image data should be discrete valued. It seems to me that any sampled output between 9.5-10.5 would all get mapped to the correct value of 10. Then it seems that you should take the term in the ELBO as
  <span class="math-container">\begin{align*} 
  \log p(x \, | \, z) = \log \big(F(10.5, \mu, \sigma^2) - F(9.5, \mu, \sigma^2)\big) \tag{2}
  \end{align*}</span>
  where <span class="math-container">$F$</span> is the CDF of the normal distribution.</p>
  <p>It seems that all references calculate the ELBO as (1) and none as (2). Why is this justified?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-04T11:45:15.257Z
Comments.Text: So you mean you want to use a (discrete) gaussian distribution to model a quantity that has finite support (i.e. {0, 1, .., 255}) ? This doesn't sound like a good design choice. Is there any reference you can provide, or is it your own design ?
Comments.UserDisplayName: ayandas
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-04T15:57:12.987Z
Comments.Text: I'm not sure you understand the question. I am describing the typical setup e.g. https://proceedings.neurips.cc/paper/2016/file/eb86d510361fc23b59f18c1bc9802cc6-Paper.pdf or the original paper https://arxiv.org/abs/1312.6114. the decoder uses a gaussian distribution. The question is asking why it is justified to use the probability density of the normal when considering that the output is an image, which as far as I understand is not technically continuous.
Comments.UserDisplayName: Taw
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-04T17:27:05.250Z
Comments.Text: I also don't understand this question, but maybe because it's been a while I had to do with VAEs. What do you mean by "compute p(x | z) using the probability density corresponding to the value of 10".
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-06T02:24:10.280Z
Comments.Text: @nbro I have edited the question to be more precise now.
Comments.UserDisplayName: Taw
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-06T12:02:41.460Z
Comments.Text: @Taw Thanks. Now, it's clearer. I've quickly checked the VAE paper and they say that they use a Gaussian and constraint the output to [0, 1] for images. Note that pixels can also been seen as having values in the range [0, 1]. Maybe this partially answers your question. Anyway, I would also advise you to put your specific question in the title now (because right now it's not a question and not specific)
Comments.UserDisplayName: nbro
Tags: <image-processing><variational-autoencoder><image-generation>
Title: variational autoencoder - decoder output for images
UserDisplayName: Taw
ThreadText: # variational autoencoder - decoder output for images
  
  (Asked by: username_0 on 2021-10-03)
  
  Following the standard setup/notation for a VAE, let $z$ denote the latent variables, $q$ as the encoder, $p$ as the decoder, and $x$ as the label. Let the objective be to maximize the ELBO, where a single sample monte carlo estimate of the ELBO is given by
  \begin{align\*}
  \log p(x \, | \, z) + \log p(z) - \log q(z \, | \, x)
  \end{align\*}
  
  Now I want to focus only on the $\log p(x \, | \, z)$ term, where $x$ is an image, and the decoder $p$ outputs the mean/variance of a normal distribution for each pixel.
  
  My understanding is that pixel values should be integer 0-255. Now consider a single pixel: suppose that the ground truth for that pixel is the value 10, and the encoder predicts the mean, variance $\mu, \sigma^2$ respectively. Now when computing the ELBO, we have this term
  \begin{align\*}
  \log p(x \, | \, z) = \log f(10, \mu, \sigma^2) \tag{1}
  \end{align\*}
  where $f$ is the probability density of the normal distribution. My question is why it is justified to compute $\log p(x \, | \, z)$ using the density considering that image data should be discrete valued. It seems to me that any sampled output between 9.5-10.5 would all get mapped to the correct value of 10. Then it seems that you should take the term in the ELBO as
  \begin{align\*}
  \log p(x \, | \, z) = \log \big(F(10.5, \mu, \sigma^2) - F(9.5, \mu, \sigma^2)\big) \tag{2}
  \end{align\*}
  where $F$ is the CDF of the normal distribution.
  
  It seems that all references calculate the ELBO as (1) and none as (2). Why is this justified?
  
  **Tags:** image-processing, variational-autoencoder, image-generation
  
  > **Comments:**
  > **username_1**: So you mean you want to use a (discrete) gaussian distribution to model a quantity that has finite support (i.e. {0, 1, .., 255}) ? This doesn't sound like a good design choice. Is there any reference you can provide, or is it your own design ?  
  > **username_0**: I'm not sure you understand the question. I am describing the typical setup e.g. https://proceedings.neurips.cc/paper/2016/file/eb86d510361fc23b59f18c1bc9802cc6-Paper.pdf or the original paper https://arxiv.org/abs/1312.6114. the decoder uses a gaussian distribution. The question is asking why it is justified to use the probability density of the normal when considering that the output is an image, which as far as I understand is not technically continuous.  
  > **username_2**: I also don't understand this question, but maybe because it's been a while I had to do with VAEs. What do you mean by "compute p(x | z) using the probability density corresponding to the value of 10".  
  > **username_0**: @username_2 I have edited the question to be more precise now.  
  > **username_2**: @username_0 Thanks. Now, it's clearer. I've quickly checked the VAE paper and they say that they use a Gaussian and constraint the output to [0, 1] for images. Note that pixels can also been seen as having values in the range [0, 1]. Maybe this partially answers your question. Anyway, I would also advise you to put your specific question in the title now (because right now it's not a question and not specific)  
  
  ---
  
  ## 0 answers
  
  