Answers.Body: <p>Indeed there's no theoretical reason you cannot get the data <span class="math-container">$\mathbf{x}_0$</span> directly from the variational noise(s) after training converges successfully based on your mentioned formula due to the Gaussian noise added to the explicitly designed fixed Markov chain of the forward diffusion process according to a certain variance schedule <span class="math-container">$\beta_{1:T}$</span>. The only reason is empirical and is explicitly mentioned in the DDPM paper's section 3.2 as follows.</p>
  <blockquote>
  <p>To summarize, we can train the reverse process mean function approximator <span class="math-container">$\mathbf{µ_θ}$</span> to predict <span class="math-container">$\mathbf{\tilde{µ}}_t$</span>, or by modifying its parameterization, we can train it to predict <span class="math-container">$\mathbf{\epsilon}$</span>. (There is also the possibility of predicting <span class="math-container">$\mathbf{x}_0$</span>, but we found this to lead to worse sample quality early in our experiments.)... Nonetheless, it is just another parameterization of <span class="math-container">$p_θ(\mathbf{x}_{t−1}|\mathbf{x}_t)$</span>, so we verify its effectiveness in Section 4 in an ablation where we compare predicting <span class="math-container">$\mathbf{\epsilon}$</span> against predicting <span class="math-container">$\mathbf{\tilde{µ}}_t$</span>.</p>
  </blockquote>
  <p>Therefore it's basically an empirical result that the reparameterization via <span class="math-container">$\mathbf{\epsilon}_{\theta}$</span> and sampling through denoising reverse process to generate data is the best algo so far by experiments. Intuitively DDPM is essentially a hierarchical latent variable model whose generative reverse process decoder is a Markov chain with learned Gaussian transitions <span class="math-container">$p_θ(\mathbf{x}_{t−1}|\mathbf{x}_t)$</span>  reparameterized via <span class="math-container">$\mathbf{\epsilon}_{\theta}$</span>, so it makes sense to follow this denoising hierarchy in the reverse process via new sampling while not to directly train from the same variational ELBO objective to predict <span class="math-container">$\mathbf{x}_0$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-20T08:13:23.963Z
Answers.LastActivityDate: 2024-07-20T08:13:23.963Z
Answers.UserDisplayName: cinch
Answers.Body: <p>From my understanding of the DDPM paper, the mistake is assuming that <span class="math-container">$\epsilon$</span> predicted by the U-net is the same <span class="math-container">$\epsilon$</span> in <span class="math-container">$$x_{t}=\sqrt{\bar\alpha_{t}}*x_{0} + \sqrt{1-\bar\alpha_{t}}*\epsilon$$</span></p>
  <p>In the DDPM paper, <span class="math-container">$$\epsilon$$</span> always refers to the noise sampled during the forward process, this is &quot;unbiased&quot; noise which is not following some learned gradient to arrive at a location in latent space, it is simply normal (no pun intended) noise meant to distort the original data.</p>
  <p><span class="math-container">$\epsilon_{\theta}$</span> is the parameterised noise which is learned and subtracted from a diffusion step in order to converge towards <span class="math-container">$X_0$</span></p>
  <p>That is why:</p>
  <p><span class="math-container">$$\epsilon_{\theta}(\sqrt{\bar{\alpha_t}}X_0+\sqrt{1-\bar{\alpha_t}}\epsilon,t)$$</span> is defined as such.</p>
  <p>The forward sampled noise is required to calculate the noised sample which the parameterised noise is predicted from.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-27T07:49:00.307Z
Answers.Text: Thanks, I can somehow understand you point. But in original DDPM paper, page 4, algorithm 1, step 4 and 5, aren't they the same ε? 
  Also in another paper https://arxiv.org/pdf/2208.11970.pdf, the whole derivation 
   in page 15, I think the ε_0 in equation-130 is the same ε_0 the author use for getting x_t from x_0 in equation-69/115.  Correct me if I made another mistake, I'm just getting more confused.
Answers.UserDisplayName: neese
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-26T10:31:11.607Z
Answers.LastActivityDate: 2023-05-26T10:33:55.347Z
Answers.LastEditDate: 2023-05-26T10:33:55.347Z
Answers.UserDisplayName: Oli_WH
Answers.Body: <p>You actually can!</p>
  <p>What you wrote makes perfect sense. It is actually given in <a href="https://hojonathanho.github.io/diffusion/assets/denoising_diffusion20.pdf" rel="nofollow noreferrer">original work</a> on page 6 equation 15.</p>
  <p>Even more, the process of how the network can reconstruct the <span class="math-container">$x_0$</span> is given in <a href="https://hojonathanho.github.io/diffusion/assets/denoising_diffusion20.pdf" rel="nofollow noreferrer">denoising diffusion paper</a> in <strong>Progressive generation</strong> section on page 7. They study exactly the process you are talking about.</p>
  <p>This process of image reconstruction is studied even more in work on <a href="https://openaccess.thecvf.com/content/CVPR2022/papers/Choi_Perception_Prioritized_Training_of_Diffusion_Models_CVPR_2022_paper.pdf" rel="nofollow noreferrer">perceptual weighted diffusion process</a>, it also can be interesting for you to look at!</p>
  <p>Hope it helps.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-20T13:25:38.577Z
Answers.LastActivityDate: 2024-02-20T13:46:34.887Z
Answers.LastEditDate: 2024-02-20T13:46:34.887Z
Answers.UserDisplayName: vl_knd
Body: <p>In DDPM, when given <span class="math-container">$x_{t}$</span> and time step <span class="math-container">$t$</span>, U-Net predicts <span class="math-container">$\epsilon$</span>, if I understand correctly, this <span class="math-container">$\epsilon$</span> is the exact same <span class="math-container">$\epsilon$</span> in equation</p>
  <p><span class="math-container">$$x_{t}=\sqrt{\bar\alpha_{t}}*x_{0} + \sqrt{1-\bar\alpha_{t}}*\epsilon $$</span></p>
  <p>Since <span class="math-container">$\bar\alpha_{t}$</span> is known, and if we have <span class="math-container">$x_{t}$</span>, we use trained model to predict <span class="math-container">$\epsilon$</span>, then why can't we get <span class="math-container">$x_{0}$</span> directly by using this equation?</p>
  
Tags: <diffusion-models>
Title: In DDPM, why can't we get $x_0$ from $x_t$ directly by using equation?
UserDisplayName: neese
ThreadText: # In DDPM, why can't we get $x_0$ from $x_t$ directly by using equation?
  
  (Asked by: username_0 on 2023-05-26)
  
  In DDPM, when given $x\_{t}$ and time step $t$, U-Net predicts $\epsilon$, if I understand correctly, this $\epsilon$ is the exact same $\epsilon$ in equation
  
  $$x\_{t}=\sqrt{\bar\alpha\_{t}}\*x\_{0} + \sqrt{1-\bar\alpha\_{t}}\*\epsilon $$
  
  Since $\bar\alpha\_{t}$ is known, and if we have $x\_{t}$, we use trained model to predict $\epsilon$, then why can't we get $x\_{0}$ directly by using this equation?
  
  **Tags:** diffusion-models
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2024-07-20. Score: 0
  
  Indeed there's no theoretical reason you cannot get the data $\mathbf{x}\_0$ directly from the variational noise(s) after training converges successfully based on your mentioned formula due to the Gaussian noise added to the explicitly designed fixed Markov chain of the forward diffusion process according to a certain variance schedule $\beta\_{1:T}$. The only reason is empirical and is explicitly mentioned in the DDPM paper's section 3.2 as follows.
  
  > To summarize, we can train the reverse process mean function approximator $\mathbf{µ\_θ}$ to predict $\mathbf{\tilde{µ}}\_t$, or by modifying its parameterization, we can train it to predict $\mathbf{\epsilon}$. (There is also the possibility of predicting $\mathbf{x}\_0$, but we found this to lead to worse sample quality early in our experiments.)... Nonetheless, it is just another parameterization of $p\_θ(\mathbf{x}\_{t−1}|\mathbf{x}\_t)$, so we verify its effectiveness in Section 4 in an ablation where we compare predicting $\mathbf{\epsilon}$ against predicting $\mathbf{\tilde{µ}}\_t$.
  
  Therefore it's basically an empirical result that the reparameterization via $\mathbf{\epsilon}\_{\theta}$ and sampling through denoising reverse process to generate data is the best algo so far by experiments. Intuitively DDPM is essentially a hierarchical latent variable model whose generative reverse process decoder is a Markov chain with learned Gaussian transitions $p\_θ(\mathbf{x}\_{t−1}|\mathbf{x}\_t)$ reparameterized via $\mathbf{\epsilon}\_{\theta}$, so it makes sense to follow this denoising hierarchy in the reverse process via new sampling while not to directly train from the same variational ELBO objective to predict $\mathbf{x}\_0$.
  
  
  ---
  
  ### Answer by username_2 on 2023-05-26. Score: 0
  
  From my understanding of the DDPM paper, the mistake is assuming that $\epsilon$ predicted by the U-net is the same $\epsilon$ in $$x\_{t}=\sqrt{\bar\alpha\_{t}}\*x\_{0} + \sqrt{1-\bar\alpha\_{t}}\*\epsilon$$
  
  In the DDPM paper, $$\epsilon$$ always refers to the noise sampled during the forward process, this is "unbiased" noise which is not following some learned gradient to arrive at a location in latent space, it is simply normal (no pun intended) noise meant to distort the original data.
  
  $\epsilon\_{\theta}$ is the parameterised noise which is learned and subtracted from a diffusion step in order to converge towards $X\_0$
  
  That is why:
  
  $$\epsilon\_{\theta}(\sqrt{\bar{\alpha\_t}}X\_0+\sqrt{1-\bar{\alpha\_t}}\epsilon,t)$$ is defined as such.
  
  The forward sampled noise is required to calculate the noised sample which the parameterised noise is predicted from.
  
  > **Comments:**
  > **username_0**: Thanks, I can somehow understand you point. But in original DDPM paper, page 4, algorithm 1, step 4 and 5, aren't they the same ε?  Also in another paper https://arxiv.org/pdf/2208.11970.pdf, the whole derivation   in page 15, I think the ε_0 in equation-130 is the same ε_0 the author use for getting x_t from x_0 in equation-69/115.  Correct me if I made another mistake, I'm just getting more confused.  
  
  ---
  
  ### Answer by username_3 on 2024-02-20. Score: 0
  
  You actually can!
  
  What you wrote makes perfect sense. It is actually given in [original work](https://hojonathanho.github.io/diffusion/assets/denoising_diffusion20.pdf) on page 6 equation 15.
  
  Even more, the process of how the network can reconstruct the $x\_0$ is given in [denoising diffusion paper](https://hojonathanho.github.io/diffusion/assets/denoising_diffusion20.pdf) in **Progressive generation** section on page 7. They study exactly the process you are talking about.
  
  This process of image reconstruction is studied even more in work on [perceptual weighted diffusion process](https://openaccess.thecvf.com/content/CVPR2022/papers/Choi_Perception_Prioritized_Training_of_Diffusion_Models_CVPR_2022_paper.pdf), it also can be interesting for you to look at!
  
  Hope it helps.
  
  
  ---
  
  