Answers.Body: <p>I have recently taken a seminar on a similar let me explain you in brief,</p>
  <p>The complete end-to-end process has 3 steps while inferencing:</p>
  <ul>
  <li>Text Encoding (using CLIP Model)</li>
  <li>Image Information Creator (Unet + Scheduler)</li>
  <li>Image Decoder (Auto Encoder)</li>
  </ul>
  <p><a href="https://i.sstatic.net/j7Ne5.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/j7Ne5.png" alt="enter image description here" /></a></p>
  <ul>
  <li>Text Encoding: These Encoding models are trained on Image to caption Database or similar, They are good in matching Text with Image data.</li>
  </ul>
  <p><a href="https://i.sstatic.net/mensT.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/mensT.png" alt="enter image description here" /></a></p>
  <ul>
  <li>Image Information Creator (Unet + Scheduler): The Unet model is trained to Predict the noisy Part of the Image.</li>
  </ul>
  <p><a href="https://i.sstatic.net/u8g3y.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/u8g3y.png" alt="enter image description here" /></a></p>
  <ul>
  <li>For the end-to-end process of Image generation. An attention module is introduced which will attend to the Text Embeddings and change the Image pixels in that direction</li>
  </ul>
  <p><a href="https://i.sstatic.net/GYSFn.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/GYSFn.png" alt="enter image description here" /></a></p>
  <ul>
  <li><p>Scheduler: Scheduler will add noise to image after each iteration for variation of pixels.
  <a href="https://i.sstatic.net/uZQHS.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/uZQHS.png" alt="enter image description here" /></a></p>
  </li>
  <li><p>VAE: All the processes mentioned above are in latent space for faster convergence and speed, VAE later converts the latent image to a pixel image.</p>
  </li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-02T13:06:30.180Z
Answers.Text: Can you provide a reference (research paper or book that describe what you're describing)?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-02T13:56:56.633Z
Answers.Text: @nbro it's from multiple sources like 
  https://jalammar.github.io/illustrated-transformer/
  https://jalammar.github.io/illustrated-stable-diffusion/
  https://stable-diffusion-art.com/how-stable-diffusion-work/
  https://medium.com/@onkarmishra/stable-diffusion-explained-1f101284484d
Answers.UserDisplayName: Hiren Namera
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-02T06:12:54.167Z
Answers.LastActivityDate: 2024-01-02T06:12:54.167Z
Answers.UserDisplayName: Hiren Namera
Body: <p>I'm interested in how text to image models like Midjourney and Dall-E work, where you enter a text prompt and get as output some images. I started reading some papers on it and stumbled upon &quot;Denoising Diffusion Probabilistic Models&quot; - <a href="https://arxiv.org/pdf/2006.11239.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/2006.11239.pdf</a>.</p>
  <p>It describes the diffusion process and how there is a sequence of random variables, <span class="math-container">$x_t$</span>, moving between them through a Markov chain, etc.</p>
  <p>But neither in this paper nor in any others did I find an explanation for how the training data (which I assume is tuples of images and accompanying text) is actually used. How does it play a role in the likelihood/ objective function?</p>
  
Tags: <image-generation><diffusion-models><midjourney>
Title: How do stable diffusion models take the data into account
UserDisplayName: Rohit Pandey
ThreadText: # How do stable diffusion models take the data into account
  
  (Asked by: username_0 on 2023-12-30)
  
  I'm interested in how text to image models like Midjourney and Dall-E work, where you enter a text prompt and get as output some images. I started reading some papers on it and stumbled upon "Denoising Diffusion Probabilistic Models" - <https://arxiv.org/pdf/2006.11239.pdf>.
  
  It describes the diffusion process and how there is a sequence of random variables, $x\_t$, moving between them through a Markov chain, etc.
  
  But neither in this paper nor in any others did I find an explanation for how the training data (which I assume is tuples of images and accompanying text) is actually used. How does it play a role in the likelihood/ objective function?
  
  **Tags:** image-generation, diffusion-models, midjourney
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-01-02. Score: 2
  
  I have recently taken a seminar on a similar let me explain you in brief,
  
  The complete end-to-end process has 3 steps while inferencing:
  
  * Text Encoding (using CLIP Model)
  * Image Information Creator (Unet + Scheduler)
  * Image Decoder (Auto Encoder)
  
  [![enter image description here](https://i.sstatic.net/j7Ne5.png)](https://i.sstatic.net/j7Ne5.png)
  
  * Text Encoding: These Encoding models are trained on Image to caption Database or similar, They are good in matching Text with Image data.
  
  [![enter image description here](https://i.sstatic.net/mensT.png)](https://i.sstatic.net/mensT.png)
  
  * Image Information Creator (Unet + Scheduler): The Unet model is trained to Predict the noisy Part of the Image.
  
  [![enter image description here](https://i.sstatic.net/u8g3y.png)](https://i.sstatic.net/u8g3y.png)
  
  * For the end-to-end process of Image generation. An attention module is introduced which will attend to the Text Embeddings and change the Image pixels in that direction
  
  [![enter image description here](https://i.sstatic.net/GYSFn.png)](https://i.sstatic.net/GYSFn.png)
  
  * Scheduler: Scheduler will add noise to image after each iteration for variation of pixels.
    [![enter image description here](https://i.sstatic.net/uZQHS.png)](https://i.sstatic.net/uZQHS.png)
  * VAE: All the processes mentioned above are in latent space for faster convergence and speed, VAE later converts the latent image to a pixel image.
  
  > **Comments:**
  > **username_2**: Can you provide a reference (research paper or book that describe what you're describing)?  
  > **username_1**: @username_2 it's from multiple sources like  https://jalammar.github.io/illustrated-transformer/ https://jalammar.github.io/illustrated-stable-diffusion/ https://stable-diffusion-art.com/how-stable-diffusion-work/ https://medium.com/@onkarmishra/stable-diffusion-explained-1f101284484d  
  
  ---
  
  