Answers.Body: <p>Intuitively, the noise schedule shall be smooth and progressive enough to be easily approximated by the NN.  In this context, a linear schedule is plausibly too &quot;easy&quot;, and hence prone to overfitting.</p>
  <p>In contrast, the modulation of the cosine schedule provides a slightly more challenging task to the NN (yet remaining smooth and gradual), so that when the reverse process is learnt, it is more robust.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-09T10:46:37.177Z
Answers.Text: Thank you for relay. I understand the overfitting. I see.
Answers.UserDisplayName: diffusion stable
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-04T16:00:21.517Z
Answers.LastActivityDate: 2023-07-04T16:00:21.517Z
Answers.UserDisplayName: Peblo
Answers.Body: <p>I will suppose that you already have understood how diffusion models work. Some good resources are this <a href="https://lilianweng.github.io/posts/2021-07-11-diffusion-models/" rel="nofollow noreferrer">blog</a> and the DDPM <a href="https://arxiv.org/abs/2006.11239" rel="nofollow noreferrer">paper</a>.</p>
  <p>If we look at Figure 3 of the paper, we see that in linear schedule the image are almost purely noise in the last quarter.
  During sampling, we usually perform the same number of steps the model was trained on, but in the linear schedule the reverse process will just turn some random noise into some other random noise for 1/4 of the time.</p>
  <p><a href="https://i.sstatic.net/Vt4yG.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Vt4yG.png" alt="Linear vs cosine schedule comparison" /></a></p>
  <ol>
  <li>If those steps are irrelevant in terms of image quality (estimated using FID), it means that we are just wasting time (and a bunch precious energy to run GPUs)</li>
  <li>The choice of the noise schedule used during training, i.e the degree with which the image is gradually turned into noise, will affect the performance (measuring the quality of generated images during sampling) of the diffusion model. If the performance at test time does not change much when skipping training steps, this means that the model has learned nothing for a fraction of his training steps, so the authors aim to find a schedule where each training step positively impacts the image quality at test time.</li>
  <li>Another proposed solution of this paper is in the learned variances <span class="math-container">$\Sigma_\theta(x,t)$</span>. The original paper sets <span class="math-container">$\Sigma_\theta(x,t)= \sigma^2 I$</span>, while this paper proposes an interpolation
  <span class="math-container">$$\Sigma_\theta(x,t)= \exp(v \log \beta_t + (1-v) \log \tilde{\beta_t})$$</span>
  In figure 8 they show that with this modification the number of sampling steps can be reduced without losing image quality (measured by FID). All but <span class="math-container">$\Sigma_\theta(x,t)= \beta^2 I$</span> obtain the same performance in 400 vs 4K denoising steps.</li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-08T01:15:11.727Z
Answers.Text: Thank you for reply. I understand about the Figure 8. On the other hand , does Figure 4 mean that the cosine schedule gets much worse when skipped up to 20%?
Answers.UserDisplayName: diffusion stable
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-08T01:22:47.473Z
Answers.Text: In the case of Figure 4, does it mean cosine schedule is not good?
Answers.UserDisplayName: diffusion stable
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-08T06:43:29.920Z
Answers.Text: @diffusionstable no, the cosine schedule is good because intuitively every denoising step is useful to produce a better image, whereas in linear schedule many steps do not contribute in improving the final result.
Answers.UserDisplayName: Ciodar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-18T06:05:34.153Z
Answers.Text: I'm sorry to append another question, but since a higher FID value indicates lower accuracy, regarding the FID figures, it is written that even if the reverse diffusion process is skipped by 20%, the linear schedule does not deteriorate. What does this mean? (It seems the linear schedule is good.)
Answers.UserDisplayName: diffusion stable
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-21T14:10:13.230Z
Answers.Text: @diffusionstable A Higher FID indicates a lower image quality. Skipping steps while mantaining the same FID is a nice property during sampling, but not during training. If some (up to 1/4th, in this case) of the steps were not useful during training, the model will likely produce lower quality images, compared to a model trained with a schedule where each step is useful for learning. You can observe this by looking at table 1, where the linear schedule achieves a higher test FID compared to the cosine schedule.
Answers.UserDisplayName: Ciodar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-03T08:23:31.643Z
Answers.LastActivityDate: 2023-09-21T14:19:02.043Z
Answers.LastEditDate: 2023-09-21T14:19:02.043Z
Answers.UserDisplayName: Ciodar
Body: <p>Could you provide the explanation of Figure 4 from the paper at <a href="https://arxiv.org/abs/2102.09672" rel="nofollow noreferrer">Improved Denoising Diffusion Probabilistic Models</a>?</p>
  <p>(1) The paper says, &quot;<em>the end of the forward noising process is too noisy,and so doesn’t contribute very much to sample quality</em>&quot;. But if the goal is to have an image with only noise, why is it problematic to have a lot of noise?</p>
  <p>(2) The paper says, &quot;<em>a model trained with the linear schedule does not get much worse (as measured by FID) when we skip up to 20% of the reverse diffusion process</em>&quot;. But why is it the reverse diffusion process? Shouldn't the deterioration be related to the forward  process?</p>
  <p>(3) Also, why is the training process relevant?</p>
  <p>It seems to me that the explanation of this section means 'linear noise schedule is better than cosine one'.
  Could you explain, please?</p>
  <p><a href="https://i.sstatic.net/4Og2w.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/4Og2w.png" alt="enter image description here" /></a></p>
  
Tags: <machine-learning><deep-learning><generative-model><diffusion-models>
Title: About cosine noise schedule in Diffusion Model
UserDisplayName: diffusion stable
ThreadText: # About cosine noise schedule in Diffusion Model
  
  (Asked by: username_0 on 2023-07-01)
  
  Could you provide the explanation of Figure 4 from the paper at [Improved Denoising username_0 Probabilistic Models](https://arxiv.org/abs/2102.09672)?
  
  (1) The paper says, "*the end of the forward noising process is too noisy,and so doesn’t contribute very much to sample quality*". But if the goal is to have an image with only noise, why is it problematic to have a lot of noise?
  
  (2) The paper says, "*a model trained with the linear schedule does not get much worse (as measured by FID) when we skip up to 20% of the reverse username_0 process*". But why is it the reverse username_0 process? Shouldn't the deterioration be related to the forward process?
  
  (3) Also, why is the training process relevant?
  
  It seems to me that the explanation of this section means 'linear noise schedule is better than cosine one'.
  Could you explain, please?
  
  [![enter image description here](https://i.sstatic.net/4Og2w.png)](https://i.sstatic.net/4Og2w.png)
  
  **Tags:** machine-learning, deep-learning, generative-model, diffusion-models
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2023-07-03. Score: 3 | Accepted answer
  
  I will suppose that you already have understood how username_0 models work. Some good resources are this [blog](https://lilianweng.github.io/posts/2021-07-11-username_0-models/) and the DDPM [paper](https://arxiv.org/abs/2006.11239).
  
  If we look at Figure 3 of the paper, we see that in linear schedule the image are almost purely noise in the last quarter.
  During sampling, we usually perform the same number of steps the model was trained on, but in the linear schedule the reverse process will just turn some random noise into some other random noise for 1/4 of the time.
  
  [![Linear vs cosine schedule comparison](https://i.sstatic.net/Vt4yG.png)](https://i.sstatic.net/Vt4yG.png)
  
  1. If those steps are irrelevant in terms of image quality (estimated using FID), it means that we are just wasting time (and a bunch precious energy to run GPUs)
  2. The choice of the noise schedule used during training, i.e the degree with which the image is gradually turned into noise, will affect the performance (measuring the quality of generated images during sampling) of the username_0 model. If the performance at test time does not change much when skipping training steps, this means that the model has learned nothing for a fraction of his training steps, so the authors aim to find a schedule where each training step positively impacts the image quality at test time.
  3. Another proposed solution of this paper is in the learned variances $\Sigma\_\theta(x,t)$. The original paper sets $\Sigma\_\theta(x,t)= \sigma^2 I$, while this paper proposes an interpolation
     $$\Sigma\_\theta(x,t)= \exp(v \log \beta\_t + (1-v) \log \tilde{\beta\_t})$$
     In figure 8 they show that with this modification the number of sampling steps can be reduced without losing image quality (measured by FID). All but $\Sigma\_\theta(x,t)= \beta^2 I$ obtain the same performance in 400 vs 4K denoising steps.
  
  > **Comments:**
  > **username_0**: Thank you for reply. I understand about the Figure 8. On the other hand , does Figure 4 mean that the cosine schedule gets much worse when skipped up to 20%?  
  > **username_0**: In the case of Figure 4, does it mean cosine schedule is not good?  
  > **username_2**: @username_0 no, the cosine schedule is good because intuitively every denoising step is useful to produce a better image, whereas in linear schedule many steps do not contribute in improving the final result.  
  > **username_0**: I'm sorry to append another question, but since a higher FID value indicates lower accuracy, regarding the FID figures, it is written that even if the reverse username_0 process is skipped by 20%, the linear schedule does not deteriorate. What does this mean? (It seems the linear schedule is good.)  
  > **username_2**: @username_0 A Higher FID indicates a lower image quality. Skipping steps while mantaining the same FID is a nice property during sampling, but not during training. If some (up to 1/4th, in this case) of the steps were not useful during training, the model will likely produce lower quality images, compared to a model trained with a schedule where each step is useful for learning. You can observe this by looking at table 1, where the linear schedule achieves a higher test FID compared to the cosine schedule.  
  
  ---
  
  ### Answer by username_1 on 2023-07-04. Score: 1
  
  Intuitively, the noise schedule shall be smooth and progressive enough to be easily approximated by the NN. In this context, a linear schedule is plausibly too "easy", and hence prone to overfitting.
  
  In contrast, the modulation of the cosine schedule provides a slightly more challenging task to the NN (yet remaining smooth and gradual), so that when the reverse process is learnt, it is more robust.
  
  > **Comments:**
  > **username_0**: Thank you for relay. I understand the overfitting. I see.  
  
  ---
  
  