<p>I have recently taken a seminar on a similar let me explain you in brief,</p>
<p>The complete end-to-end process has 3 steps while inferencing:</p>
<ul>
<li>Text Encoding (using CLIP Model)</li>
<li>Image Information Creator (Unet + Scheduler)</li>
<li>Image Decoder (Auto Encoder)</li>
</ul>
<p><a href="https://i.sstatic.net/j7Ne5.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/j7Ne5.png" alt="enter image description here" /></a></p>
<ul>
<li>Text Encoding: These Encoding models are trained on Image to caption Database or similar, They are good in matching Text with Image data.</li>
</ul>
<p><a href="https://i.sstatic.net/mensT.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/mensT.png" alt="enter image description here" /></a></p>
<ul>
<li>Image Information Creator (Unet + Scheduler): The Unet model is trained to Predict the noisy Part of the Image.</li>
</ul>
<p><a href="https://i.sstatic.net/u8g3y.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/u8g3y.png" alt="enter image description here" /></a></p>
<ul>
<li>For the end-to-end process of Image generation. An attention module is introduced which will attend to the Text Embeddings and change the Image pixels in that direction</li>
</ul>
<p><a href="https://i.sstatic.net/GYSFn.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/GYSFn.png" alt="enter image description here" /></a></p>
<ul>
<li><p>Scheduler: Scheduler will add noise to image after each iteration for variation of pixels.
<a href="https://i.sstatic.net/uZQHS.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/uZQHS.png" alt="enter image description here" /></a></p>
</li>
<li><p>VAE: All the processes mentioned above are in latent space for faster convergence and speed, VAE later converts the latent image to a pixel image.</p>
</li>
</ul>

CC BY-SA 4.0
2024-01-02T13:06:30.180Z
Can you provide a reference (research paper or book that describe what you're describing)?
nbro
CC BY-SA 4.0
2024-01-02T13:56:56.633Z
@nbro it's from multiple sources like 
https://jalammar.github.io/illustrated-transformer/
https://jalammar.github.io/illustrated-stable-diffusion/
https://stable-diffusion-art.com/how-stable-diffusion-work/
https://medium.com/@onkarmishra/stable-diffusion-explained-1f101284484d
Hiren Namera
CC BY-SA 4.0
2024-01-02T06:12:54.167Z
2024-01-02T06:12:54.167Z
Hiren Namera
<p>I'm interested in how text to image models like Midjourney and Dall-E work, where you enter a text prompt and get as output some images. I started reading some papers on it and stumbled upon &quot;Denoising Diffusion Probabilistic Models&quot; - <a href="https://arxiv.org/pdf/2006.11239.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/2006.11239.pdf</a>.</p>
<p>It describes the diffusion process and how there is a sequence of random variables, <span class="math-container">$x_t$</span>, moving between them through a Markov chain, etc.</p>
<p>But neither in this paper nor in any others did I find an explanation for how the training data (which I assume is tuples of images and accompanying text) is actually used. How does it play a role in the likelihood/ objective function?</p>

<image-generation><diffusion-models><midjourney>
How do stable diffusion models take the data into account
Rohit Pandey
# How do stable diffusion models take the data into account

(Asked by: username_0 on 2023-12-30)

I'm interested in how text to image models like Midjourney and Dall-E work, where you enter a text prompt and get as output some images. I started reading some papers on it and stumbled upon "Denoising Diffusion Probabilistic Models" - <https://arxiv.org/pdf/2006.11239.pdf>.

It describes the diffusion process and how there is a sequence of random variables, $x\_t$, moving between them through a Markov chain, etc.

But neither in this paper nor in any others did I find an explanation for how the training data (which I assume is tuples of images and accompanying text) is actually used. How does it play a role in the likelihood/ objective function?

**Tags:** image-generation, diffusion-models, midjourney


---

## 1 answers

### Answer by username_1 on 2024-01-02. Score: 2

I have recently taken a seminar on a similar let me explain you in brief,

The complete end-to-end process has 3 steps while inferencing:

* Text Encoding (using CLIP Model)
* Image Information Creator (Unet + Scheduler)
* Image Decoder (Auto Encoder)

[![enter image description here](https://i.sstatic.net/j7Ne5.png)](https://i.sstatic.net/j7Ne5.png)

* Text Encoding: These Encoding models are trained on Image to caption Database or similar, They are good in matching Text with Image data.

[![enter image description here](https://i.sstatic.net/mensT.png)](https://i.sstatic.net/mensT.png)

* Image Information Creator (Unet + Scheduler): The Unet model is trained to Predict the noisy Part of the Image.

[![enter image description here](https://i.sstatic.net/u8g3y.png)](https://i.sstatic.net/u8g3y.png)

* For the end-to-end process of Image generation. An attention module is introduced which will attend to the Text Embeddings and change the Image pixels in that direction

[![enter image description here](https://i.sstatic.net/GYSFn.png)](https://i.sstatic.net/GYSFn.png)

* Scheduler: Scheduler will add noise to image after each iteration for variation of pixels.
  [![enter image description here](https://i.sstatic.net/uZQHS.png)](https://i.sstatic.net/uZQHS.png)
* VAE: All the processes mentioned above are in latent space for faster convergence and speed, VAE later converts the latent image to a pixel image.

> **Comments:**
> **username_2**: Can you provide a reference (research paper or book that describe what you're describing)?  
> **username_1**: @username_2 it's from multiple sources like  https://jalammar.github.io/illustrated-transformer/ https://jalammar.github.io/illustrated-stable-diffusion/ https://stable-diffusion-art.com/how-stable-diffusion-work/ https://medium.com/@onkarmishra/stable-diffusion-explained-1f101284484d  

---

