Answers.Body: <p>In a way, you're right. The reconstruction loss is just an idea because you have not yet defined the distribution <span class="math-container">$p_\theta$</span>. If you assume that this distribution is e.g. a Gaussian, then you should be able to derive something interesting. See equation 12 from <a href="https://arxiv.org/pdf/1312.6114.pdf" rel="nofollow noreferrer">the VAE paper</a>. You can check out the definition of the Gaussian pdf <a href="https://en.wikipedia.org/wiki/Normal_distribution" rel="nofollow noreferrer">on Wikipedia</a>. If you're stuck, you could start with the simple regression problem and, if necessary, watch e.g. <a href="https://www.youtube.com/watch?v=vEPQNwxd1Y4" rel="nofollow noreferrer">this Hinton's video</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-04T00:47:39.220Z
Answers.LastActivityDate: 2023-08-04T00:54:34.177Z
Answers.LastEditDate: 2023-08-04T00:54:34.177Z
Answers.UserDisplayName: nbro
Answers.Body: <p>If you model the distribution <span class="math-container">$p_\theta$</span> as Gaussian then:</p>
  <p><span class="math-container">$\displaystyle p_\theta(x|z) = \frac{1}{\sigma\sqrt{2\pi}}e^{-\frac{1}{2}\frac{x-\mu}{\sigma}^2} $</span></p>
  <p><span class="math-container">$\displaystyle \log p_\theta(x|z) = -log \sigma - \frac{1}{2}\log 2\pi -\frac{1}{2} \frac{x-\mu}{\sigma}^2$</span></p>
  <p>If you decide that your neural net will only produce the mean <span class="math-container">$\mu$</span> and you will use a constant <span class="math-container">$\sigma$</span>, then the derivative of <span class="math-container">$\log p_\theta$</span> is the same as the derivative of <span class="math-container">$MSE$</span>, but scaled with <span class="math-container">$\sigma$</span>.</p>
  <p>Now, given that this is the most common choice of model (Gaussian with constant <span class="math-container">$\sigma$</span>), people just optimize the <span class="math-container">$MSE$</span> and adjust their learning rate accordingly.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-04T14:22:45.673Z
Answers.Text: Is that $ - \log\sigma - \frac{1}{2} \log 2\pi - \frac{1}{2} \Big( \frac{x-\mu}{\sigma} \Big)^2 $ ?
Answers.UserDisplayName: diffusion stable
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-04T09:59:15.550Z
Answers.LastActivityDate: 2023-08-05T08:03:36.550Z
Answers.LastEditDate: 2023-08-05T08:03:36.550Z
Answers.UserDisplayName: pi-tau
Body: <p>How to expand <b>reconstruction error</b> to <b>mean squared error</b> when it is <span class="math-container">$\mathbb{E}_{z\sim q_{\phi}(z|x)}[\log p_\theta(x|z)]$</span>?</p>
  <p>[reconstruction error]<br>
  <span class="math-container">$\mathbb{E}_{z\sim q_{\phi}(z|x)}[\log p_\theta(x|z)]$</span></p>
  <p>[mean square error]<br>
  <span class="math-container">$\mathbb{E} \Big [\big(x - p_\theta(q_\phi(x))\big)^2  \Big]$</span></p>
  <p>[MSE pseudo code]</p>
  <pre><code>def reconstruction_loss(y, t):
      # MSE y:predicted value
      #     t:true value
      return square(y - t).mean()
  </code></pre>
  <p>Is the <b>reconstruction error</b> just an idea and is actual formula the <b>MSE</b>? Is that all?</p>
  <p>I mean, why does MSE (or BCE) comes from reconstruction error?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-03T13:20:58.450Z
Comments.Text: I see 2 problems/questions here (reconstruction to MSE and reconstruction to BCE). Could you please focus on 1 problem at a time? You should create a separate post for the 2nd question. Thanks. You should also provide more details about why you want to do this and why you're stuck at doing it or what your ideas are.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-03T17:12:39.383Z
Comments.Text: I edited the question clearly. Thank you.
Comments.UserDisplayName: diffusion stable
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-04T00:42:53.477Z
Comments.Text: Thanks. Can you please now provide the source of the MSE you're showing us? This isn't from the original paper. Something doesn't seem to be right in that formula.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-04T07:43:24.103Z
Comments.Text: Yes. I Added pseudo code of MSE.
Comments.UserDisplayName: diffusion stable
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-04T10:29:27.217Z
Comments.Text: Yes, the formula is not consistent with the pseudocode. The parentheses are not correctly placed. You need to square the subtraction according to the pseudocode
Comments.UserDisplayName: nbro
Tags: <machine-learning><variational-autoencoder><diffusion-models>
Title: How to expand reconstruction error to mean squared error in Variational AutoEncoder?
UserDisplayName: diffusion stable
ThreadText: # How to expand reconstruction error to mean squared error in Variational AutoEncoder?
  
  (Asked by: username_0 on 2023-08-03)
  
  How to expand **reconstruction error** to **mean squared error** when it is $\mathbb{E}\_{z\sim q\_{\phi}(z|x)}[\log p\_\theta(x|z)]$?
  
  [reconstruction error]  
  $\mathbb{E}\_{z\sim q\_{\phi}(z|x)}[\log p\_\theta(x|z)]$
  
  [mean square error]  
  $\mathbb{E} \Big [\big(x - p\_\theta(q\_\phi(x))\big)^2 \Big]$
  
  [MSE pseudo code]
  
  ```
  def reconstruction_loss(y, t):
      # MSE y:predicted value
      #     t:true value
      return square(y - t).mean()
  
  ```
  
  Is the **reconstruction error** just an idea and is actual formula the **MSE**? Is that all?
  
  I mean, why does MSE (or BCE) comes from reconstruction error?
  
  **Tags:** machine-learning, variational-autoencoder, diffusion-models
  
  > **Comments:**
  > **username_1**: I see 2 problems/questions here (reconstruction to MSE and reconstruction to BCE). Could you please focus on 1 problem at a time? You should create a separate post for the 2nd question. Thanks. You should also provide more details about why you want to do this and why you're stuck at doing it or what your ideas are.  
  > **username_0**: I edited the question clearly. Thank you.  
  > **username_1**: Thanks. Can you please now provide the source of the MSE you're showing us? This isn't from the original paper. Something doesn't seem to be right in that formula.  
  > **username_0**: Yes. I Added pseudo code of MSE.  
  > **username_1**: Yes, the formula is not consistent with the pseudocode. The parentheses are not correctly placed. You need to square the subtraction according to the pseudocode  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2023-08-04. Score: 3
  
  In a way, you're right. The reconstruction loss is just an idea because you have not yet defined the distribution $p\_\theta$. If you assume that this distribution is e.g. a Gaussian, then you should be able to derive something interesting. See equation 12 from [the VAE paper](https://arxiv.org/pdf/1312.6114.pdf). You can check out the definition of the Gaussian pdf [on Wikipedia](https://en.wikipedia.org/wiki/Normal_distribution). If you're stuck, you could start with the simple regression problem and, if necessary, watch e.g. [this Hinton's video](https://www.youtube.com/watch?v=vEPQNwxd1Y4).
  
  
  ---
  
  ### Answer by username_2 on 2023-08-04. Score: 2
  
  If you model the distribution $p\_\theta$ as Gaussian then:
  
  $\displaystyle p\_\theta(x|z) = \frac{1}{\sigma\sqrt{2\pi}}e^{-\frac{1}{2}\frac{x-\mu}{\sigma}^2} $
  
  $\displaystyle \log p\_\theta(x|z) = -log \sigma - \frac{1}{2}\log 2\pi -\frac{1}{2} \frac{x-\mu}{\sigma}^2$
  
  If you decide that your neural net will only produce the mean $\mu$ and you will use a constant $\sigma$, then the derivative of $\log p\_\theta$ is the same as the derivative of $MSE$, but scaled with $\sigma$.
  
  Now, given that this is the most common choice of model (Gaussian with constant $\sigma$), people just optimize the $MSE$ and adjust their learning rate accordingly.
  
  > **Comments:**
  > **username_0**: Is that $ - \log\sigma - \frac{1}{2} \log 2\pi - \frac{1}{2} \Big( \frac{x-\mu}{\sigma} \Big)^2 $ ?  
  
  ---
  
  