Answers.Body: <p>You can think as if the network learns the gradient of the data distribution...</p>
  <p>For example, think about having some points in 1D which are distributed as 2 Gaussians:</p>
  <p><a href="https://i.sstatic.net/jvxVF.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/jvxVF.png" alt="enter image description here" /></a></p>
  <p>Learning the gradient means that if you start from a low probability mass area, you know in what direction you have to go, by performing:
  <span class="math-container">$$
  x_{t+1} = x_t + \alpha \nabla_x p(x)
  $$</span></p>
  <p>For example:</p>
  <p><a href="https://i.sstatic.net/AUEBm.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/AUEBm.png" alt="enter image description here" /></a></p>
  <p>the red arrow is the gradient, saying that if you want to go to an are that has higher probability mass, you have to go on the right, so you move your dot slightly on the right</p>
  <p>Now, this was in a 1D very simplistic scenario, but the same applies to DM, you start with a &quot;prior&quot; and then from there you learn in which direction you have to move your image to get a more likely (which in out case means &quot;more realistic&quot;) sample</p>
  <p>As a resource, I would suggest you to look at this blogpost: <a href="https://lilianweng.github.io/posts/2021-07-11-diffusion-models/" rel="nofollow noreferrer">https://lilianweng.github.io/posts/2021-07-11-diffusion-models/</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-07T14:39:06.270Z
Answers.Text: The resource link seems not working.
Answers.UserDisplayName: hanugm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-08T13:16:29.900Z
Answers.Text: Oh, now it is accessible.
Answers.UserDisplayName: hanugm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-07T14:33:52.320Z
Answers.LastActivityDate: 2023-08-07T14:33:52.320Z
Answers.UserDisplayName: Alberto
Body: <p>I've been reading a research paper titled <a href="https://openaccess.thecvf.com/content/CVPR2022/papers/Rombach_High-Resolution_Image_Synthesis_With_Latent_Diffusion_Models_CVPR_2022_paper.pdf" rel="nofollow noreferrer">High-Resolution Image Synthesis with Latent Diffusion Models</a> by <em>Robin Rombach et al.</em> and came across an a concept related to diffusion models (DMs). In the abstract, the authors state:</p>
  <blockquote>
  <p>By decomposing the image formation process into a sequential
  application of denoising autoencoders, diffusion models (DMs) achieve
  state-of-the-art synthesis results on image data and beyond.
  Additionally, <strong>their formulation allows for a guiding mechanism</strong> to
  control the image generation process without retraining. However,
  since these models typically operate directly in pixel space,
  optimization of powerful DMs often consumes hundreds of GPU days and
  inference is expensive due to sequential evaluations.</p>
  </blockquote>
  <p>I'm trying to gain a deeper understanding of this highlighted part, particularly how the formulation of DMs allows for a guidance mechanism that controls the image generation process without the need for retraining.</p>
  <p>Can anyone suggest detailed resources, or perhaps elaborate on the mechanism through which this control is achieved in DMs?</p>
  
Tags: <papers><resource-request><diffusion-models>
Title: In-depth understanding of formulation and guidance mechanisms in Diffusion models
UserDisplayName: hanugm
ThreadText: # In-depth understanding of formulation and guidance mechanisms in Diffusion models
  
  (Asked by: username_0 on 2023-08-07)
  
  I've been reading a research paper titled [High-Resolution Image Synthesis with Latent Diffusion Models](https://openaccess.thecvf.com/content/CVPR2022/papers/Rombach_High-Resolution_Image_Synthesis_With_Latent_Diffusion_Models_CVPR_2022_paper.pdf) by *Robin Rombach et al.* and came across an a concept related to diffusion models (DMs). In the abstract, the authors state:
  
  > By decomposing the image formation process into a sequential
  > application of denoising autoencoders, diffusion models (DMs) achieve
  > state-of-the-art synthesis results on image data and beyond.
  > Additionally, **their formulation allows for a guiding mechanism** to
  > control the image generation process without retraining. However,
  > since these models typically operate directly in pixel space,
  > optimization of powerful DMs often consumes hundreds of GPU days and
  > inference is expensive due to sequential evaluations.
  
  I'm trying to gain a deeper understanding of this highlighted part, particularly how the formulation of DMs allows for a guidance mechanism that controls the image generation process without the need for retraining.
  
  Can anyone suggest detailed resources, or perhaps elaborate on the mechanism through which this control is achieved in DMs?
  
  **Tags:** papers, resource-request, diffusion-models
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-08-07. Score: 2
  
  You can think as if the network learns the gradient of the data distribution...
  
  For example, think about having some points in 1D which are distributed as 2 Gaussians:
  
  [![enter image description here](https://i.sstatic.net/jvxVF.png)](https://i.sstatic.net/jvxVF.png)
  
  Learning the gradient means that if you start from a low probability mass area, you know in what direction you have to go, by performing:
  $$
  x\_{t+1} = x\_t + \alpha \nabla\_x p(x)
  $$
  
  For example:
  
  [![enter image description here](https://i.sstatic.net/AUEBm.png)](https://i.sstatic.net/AUEBm.png)
  
  the red arrow is the gradient, saying that if you want to go to an are that has higher probability mass, you have to go on the right, so you move your dot slightly on the right
  
  Now, this was in a 1D very simplistic scenario, but the same applies to DM, you start with a "prior" and then from there you learn in which direction you have to move your image to get a more likely (which in out case means "more realistic") sample
  
  As a resource, I would suggest you to look at this blogpost: <https://lilianweng.github.io/posts/2021-07-11-diffusion-models/>
  
  > **Comments:**
  > **username_0**: The resource link seems not working.  
  > **username_0**: Oh, now it is accessible.  
  
  ---
  
  