Answers.Body: <p>You can generate a noise <span class="math-container">$\epsilon_t$</span> by sampling a distribution (which you already explicitly stated as <span class="math-container">$\mathcal{N}(\mathbf{x}_t; \sqrt{1-\beta_t}\mathbf{x}_{t-1}, \beta_t\mathbf{I})$</span>) during the <em>sampling phase</em>. The &quot;origin&quot; is the standard normal distribution <span class="math-container">$\mathcal{N}(\mathbf{0}, \mathbf{I})$</span>, which is a constant instead of a variable ground truth to learn.</p>
  <p>The variable ground truth to learn (using a U-Net) during the <em>training phase</em> is rather the noise <span class="math-container">$\epsilon_t$</span> added during the sampling phase. Reverse to the diffusion (sampling) process, the noise <span class="math-container">$\epsilon_t$</span>, when subtracted from the noisier <span class="math-container">$\mathbf{x}_{t}$</span>, can produce a less noisy <span class="math-container">$\mathbf{x}_{t+1}$</span>. Therefore, <strong>given <span class="math-container">$\mathbf{x}_{t}$</span></strong>, the noise <span class="math-container">$\epsilon_t$</span> encodes certain information regarding the input data <span class="math-container">$\mathbf{x}_{T}$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-25T09:03:08.720Z
Answers.Text: Can you maybe elaborate on that? I still do not understand how the origin is determined, is it the distribution of the input data point during the specific time point?
Answers.UserDisplayName: Mariusmarten
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-09T19:23:03.097Z
Answers.Text: @Mariusmarten I edited the answer. Maybe you confuse origin with ground truth. Diffusion models are intuitively different from traditional neural networks due to the additional *sampling phase*.
Answers.UserDisplayName: lkjsfkshd
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-19T20:16:33.370Z
Answers.LastActivityDate: 2022-12-09T19:18:57.143Z
Answers.LastEditDate: 2022-12-09T19:18:57.143Z
Answers.UserDisplayName: lkjsfkshd
Body: <p>The inputs are decayed towards the origin using this formula within Denoising Diffusion Probabilistic Models (DDPMs):</p>
  <p><span class="math-container">$$q\left(\mathbf{x}_{1: T} \mid \mathbf{x}_0\right):=\prod_{t=1}^T q\left(\mathbf{x}_t \mid \mathbf{x}_{t-1}\right), \quad q\left(\mathbf{x}_t \mid \mathbf{x}_{t-1}\right):=\mathcal{N}\left(\mathbf{x}_t ; \sqrt{1-\beta_t} \mathbf{x}_{t-1}, \beta_t \mathbf{I}\right)$$</span></p>
  <p>I however do not understand how the origin is determined. How is the noise in the forward process or diffusion process computed? In the original <a href="https://arxiv.org/pdf/2006.11239.pdf" rel="nofollow noreferrer">DDPM paper</a> it is only stated that:</p>
  <blockquote>
  <p>the forward process variances <span class="math-container">$β_t$</span> can be learned by reparameterization [33] or held constant as hyperparameters.</p>
  </blockquote>
  
Tags: <generative-model><image-generation>
Title: How is the noise in the forward process in Denoising Diffusion Probabilistic Models computed?
UserDisplayName: Mariusmarten
ThreadText: # How is the noise in the forward process in Denoising Diffusion Probabilistic Models computed?
  
  (Asked by: username_0 on 2022-09-12)
  
  The inputs are decayed towards the origin using this formula within Denoising Diffusion Probabilistic Models (DDPMs):
  
  $$q\left(\mathbf{x}\_{1: T} \mid \mathbf{x}\_0\right):=\prod\_{t=1}^T q\left(\mathbf{x}\_t \mid \mathbf{x}\_{t-1}\right), \quad q\left(\mathbf{x}\_t \mid \mathbf{x}\_{t-1}\right):=\mathcal{N}\left(\mathbf{x}\_t ; \sqrt{1-\beta\_t} \mathbf{x}\_{t-1}, \beta\_t \mathbf{I}\right)$$
  
  I however do not understand how the origin is determined. How is the noise in the forward process or diffusion process computed? In the original [DDPM paper](https://arxiv.org/pdf/2006.11239.pdf) it is only stated that:
  
  > the forward process variances $β\_t$ can be learned by reparameterization [33] or held constant as hyperparameters.
  
  **Tags:** generative-model, image-generation
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-11-19. Score: 0
  
  You can generate a noise $\epsilon\_t$ by sampling a distribution (which you already explicitly stated as $\mathcal{N}(\mathbf{x}\_t; \sqrt{1-\beta\_t}\mathbf{x}\_{t-1}, \beta\_t\mathbf{I})$) during the *sampling phase*. The "origin" is the standard normal distribution $\mathcal{N}(\mathbf{0}, \mathbf{I})$, which is a constant instead of a variable ground truth to learn.
  
  The variable ground truth to learn (using a U-Net) during the *training phase* is rather the noise $\epsilon\_t$ added during the sampling phase. Reverse to the diffusion (sampling) process, the noise $\epsilon\_t$, when subtracted from the noisier $\mathbf{x}\_{t}$, can produce a less noisy $\mathbf{x}\_{t+1}$. Therefore, **given $\mathbf{x}\_{t}$**, the noise $\epsilon\_t$ encodes certain information regarding the input data $\mathbf{x}\_{T}$.
  
  > **Comments:**
  > **username_0**: Can you maybe elaborate on that? I still do not understand how the origin is determined, is it the distribution of the input data point during the specific time point?  
  > **username_1**: @username_0 I edited the answer. Maybe you confuse origin with ground truth. Diffusion models are intuitively different from traditional neural networks due to the additional *sampling phase*.  
  
  ---
  
  