Answers.Body: <p>The paper you are referencing actually already achieving text-to-image generation.</p>
  <p>It's a tricky question in my opinion, since the process that is modelled by neural network during denoising process is in fact a random stochastic process so there is no direct map between text embedding and resulting image. Actually diffusion process is a way to sample a &quot;point&quot; ( image in this case ) from conditional distribution (where the condition is a text input ).</p>
  <p>But since neural networks are not stochastic in nature, the model itself is in fact a mapping in a sense. It's not bijective, so it's not a one-to-one map, but rather an injective map, so its <strong>many-to-one map</strong>. Since you can construct two inputs for simple feedforward layer such that it gives the same output. However <strong>it is not stochastic</strong>, that is if you give it the same inputs it will produce the same output every time.</p>
  <p>The &quot;randomness&quot; or &quot;generative&quot; power of the models comes from the fact that each input is actually slightly different. Usually its common mechanism for all generative networks, the input variable is sampled from some known distribution ( often its normal gaussian ). So if you would give the model some prompt, to generate an image, something like the following will happened:</p>
  <ul>
  <li>Create embedding for the prompt ~ <span class="math-container">$x_{\text{text}}$</span></li>
  <li>Sample random variable <span class="math-container">$z \sim \mathcal{N}(0, 1)$</span>.</li>
  <li>Concatenate them together to obtain the input <span class="math-container">$x = [x_{\text{text}}, z]$</span></li>
  <li>Run the neural network to create an associated image: <span class="math-container">$\text{DenoisingDiffusion}(x)$</span></li>
  </ul>
  <p>So if in my toy example if <span class="math-container">$z$</span> is always the same value, the resulting image will actually be the same.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-16T13:47:14.443Z
Answers.LastActivityDate: 2024-02-16T13:47:14.443Z
Answers.UserDisplayName: vl_knd
Answers.Body: <p>I'm only familiar with Stable Diffusion 1.x models, but I assume that all LDMs work fundamentally the same way. In my understanding the mapping from input text to output image is a <strong>deterministic function</strong>, when you take into account the used random seed. The starting state is determined by white noise in the latent space, which is then denoised and is conditioned with the input text.</p>
  <p>Ah, vl_knd posted a good answer already.</p>
  <p>Note that the parallel GPU implementation isn't 100% deterministic, even using the same seed you may get slightly different results. This happens because floating point operations aren't associative.</p>
  <p>Edit: see some examples <a href="https://ai.stackexchange.com/a/39823/32722">here</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-16T13:50:25.027Z
Answers.LastActivityDate: 2024-02-16T13:59:21.593Z
Answers.LastEditDate: 2024-02-16T13:59:21.593Z
Answers.UserDisplayName: NikoNyrh
Body: <p>I've been exploring the capabilities of the Diffusion based text-to-image models and am curious about its underlying mathematical framework. Specifically, I'm interested in understanding how the model handles the mapping from textual inputs to image outputs. <strong>Is this process considered a function, where each input is associated with a unique output, or is it a relation, allowing for multiple outputs from the same input due to the stochastic nature of the model</strong>? Where exactly the stochasticity lies in incase of relation and what is the nature of the relation ('many to many' or 'one to many')?</p>
  <p>Consider the paper titled <strong><a href="https://arxiv.org/pdf/2112.10752.pdf" rel="nofollow noreferrer">High-Resolution Image Synthesis with Latent Diffusion Models</a></strong> by <em>Robin Rombach et.al.</em> as the underlying diffusion model for this question.</p>
  
Tags: <math><diffusion-models><text-to-image-generation>
Title: How does diffusion based text-to-image generation models Mathematically classify inputs to outputs?
UserDisplayName: hanugm
ThreadText: # How does diffusion based text-to-image generation models Mathematically classify inputs to outputs?
  
  (Asked by: username_0 on 2024-02-09)
  
  I've been exploring the capabilities of the Diffusion based text-to-image models and am curious about its underlying mathematical framework. Specifically, I'm interested in understanding how the model handles the mapping from textual inputs to image outputs. **Is this process considered a function, where each input is associated with a unique output, or is it a relation, allowing for multiple outputs from the same input due to the stochastic nature of the model**? Where exactly the stochasticity lies in incase of relation and what is the nature of the relation ('many to many' or 'one to many')?
  
  Consider the paper titled **[High-Resolution Image Synthesis with Latent Diffusion Models](https://arxiv.org/pdf/2112.10752.pdf)** by *Robin Rombach et.al.* as the underlying diffusion model for this question.
  
  **Tags:** math, diffusion-models, text-to-image-generation
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2024-02-16. Score: 1
  
  The paper you are referencing actually already achieving text-to-image generation.
  
  It's a tricky question in my opinion, since the process that is modelled by neural network during denoising process is in fact a random stochastic process so there is no direct map between text embedding and resulting image. Actually diffusion process is a way to sample a "point" ( image in this case ) from conditional distribution (where the condition is a text input ).
  
  But since neural networks are not stochastic in nature, the model itself is in fact a mapping in a sense. It's not bijective, so it's not a one-to-one map, but rather an injective map, so its **many-to-one map**. Since you can construct two inputs for simple feedforward layer such that it gives the same output. However **it is not stochastic**, that is if you give it the same inputs it will produce the same output every time.
  
  The "randomness" or "generative" power of the models comes from the fact that each input is actually slightly different. Usually its common mechanism for all generative networks, the input variable is sampled from some known distribution ( often its normal gaussian ). So if you would give the model some prompt, to generate an image, something like the following will happened:
  
  * Create embedding for the prompt ~ $x\_{\text{text}}$
  * Sample random variable $z \sim \mathcal{N}(0, 1)$.
  * Concatenate them together to obtain the input $x = [x\_{\text{text}}, z]$
  * Run the neural network to create an associated image: $\text{DenoisingDiffusion}(x)$
  
  So if in my toy example if $z$ is always the same value, the resulting image will actually be the same.
  
  
  ---
  
  ### Answer by username_2 on 2024-02-16. Score: 1
  
  I'm only familiar with Stable Diffusion 1.x models, but I assume that all LDMs work fundamentally the same way. In my understanding the mapping from input text to output image is a **deterministic function**, when you take into account the used random seed. The starting state is determined by white noise in the latent space, which is then denoised and is conditioned with the input text.
  
  Ah, vl\_knd posted a good answer already.
  
  Note that the parallel GPU implementation isn't 100% deterministic, even using the same seed you may get slightly different results. This happens because floating point operations aren't associative.
  
  Edit: see some examples [here](https://ai.stackexchange.com/a/39823/32722).
  
  
  ---
  
  