Answers.Body: <p>There is an intuitive reason if you look past the logarithm and the score.</p>
  <p>The fact is that classifier-free guidance is really a geometric average between the conditional and unconditional probabilities. Basically,</p>
  <p><span class="math-container">$$p(x \mid y) \propto p(x \mid y)^{\gamma}p(x)^{1 - \gamma}$$</span></p>
  <p>Why is this valid? In the limiting case, consider a scenario where <span class="math-container">$x$</span> is actually independent of <span class="math-container">$y$</span>, i.e. <span class="math-container">$p(x \mid y) = p(x)$</span>. In that case,</p>
  <p><span class="math-container">$$
  p(x \mid y) \propto p(x \mid y)^{\gamma}p(x)^{1 - \gamma} = p(x)^\gamma p(x)^{1 - \gamma} \propto p(x)
  $$</span></p>
  <p>It is consistent.</p>
  <p>However, if you consider arbitrary scaling factors <span class="math-container">$\gamma_1$</span> and <span class="math-container">$-\gamma_2$</span>, you lose the geometric consistency. Effectively, what is going on is:</p>
  <p><span class="math-container">$$
  p(x \mid y) ^{\gamma_1}p(x)^{-\gamma_2} \propto p(x \mid y)^{\tau}
  $$</span></p>
  <p>Where <span class="math-container">$\tau$</span> is some constant. The limiting case where <span class="math-container">$x$</span> and <span class="math-container">$y$</span> are independent makes this obvious, you get <span class="math-container">$p(x)^{\gamma_1 - \gamma_2}$</span>. So as long as <span class="math-container">$\gamma_2 \neq \gamma_1 - 1$</span>, you are sampling from some <span class="math-container">$p(x \mid y)^\tau$</span> (which is usually strange stuff way off the data manifold), while you really want to just sample from <span class="math-container">$p(x \mid y)^1 = p(x \mid y)$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-08T23:55:00.173Z
Answers.LastActivityDate: 2024-11-08T23:55:00.173Z
Answers.UserDisplayName: Mercury
Answers.Body: <p><strong>You probably <em>could</em> express the guidance score in that form, but it's important understand the intention behind how things are presented.</strong></p>
  <p>In this context, we only really care about the <em>relative</em> magnitude of the conditional and unconditional terms of the guidance score. When controlling the generation of images with class conditioning, it wouldn't matter how large the conditional term is in isolation if it's overshadowed by unconditional term.</p>
  <p>By writing the guidance this way, we think about the score in terms of which signal (either conditional or unconditional) is <em>dominating</em> the overall feedback to the model. This &quot;level of domination&quot; only needs one knob to express (<span class="math-container">$\gamma$</span>).</p>
  <p>If you were writing this paper, you <em>could</em> add an extra scaling term around the guidance score, adjusting the <em>overall</em> magnitude of the feedback and achieve the same degrees of freedom as your suggested formulation. Looking at the <a href="https://arxiv.org/pdf/2207.12598.pdf" rel="nofollow noreferrer">original classifier-free guidance paper</a>, it seems like there is a similar hyperparameter that you could adjust, although it's not exactly the same.</p>
  <p>However, you could just as reasonably omit this term as (1) it would require extra space to explain, but it (2) doesn't change the formula significantly, and (3) is an adjustment that readers could reasonably arrive to themselves, if they <em>do</em> end up wanting this extra flexibility.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-10T12:22:54.390Z
Answers.Text: First of all, I must again state that, from my experiments using huggingface models, using any parameter other than $\gamma_{1} = \gamma_{2} + 1$ yields awful results (not even a normal picture), so I'm not sure your answer is correct.
  Secondly, what extra scaling hyperparameter are you talking about in the original CFG paper?
Answers.UserDisplayName: snatchysquid
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-10T21:17:54.993Z
Answers.Text: @snatchysquid What values were you trying that worked/didn't work? Can you also link to the models that you were using/where you were making the changes?
Answers.UserDisplayName: Alexander Wan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-11T09:44:45.527Z
Answers.Text: I used for example the model in the following colab [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers_doc/en/pytorch/write_own_pipeline.ipynb) and simply change the Classifier-Free Guidance scale. choosing even a difference of 1.25 (instead of 1) between $\gamma_{1},\gamma_{2}$ resulted in either a blank (brownish) image, or a blank image with a small figure that sometimes matches the prompt (e.g. a banana riding on a skateboard and wearing a hat)
Answers.UserDisplayName: snatchysquid
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-10T08:31:25.670Z
Answers.LastActivityDate: 2023-09-10T09:17:35.450Z
Answers.LastEditDate: 2023-09-10T09:17:35.450Z
Answers.UserDisplayName: Alexander Wan
Body: <h2>Background | Classifier-Free Guidance Derivation</h2>
  <hr />
  <p>To summarize the derivation of Classifier-Free Guidance, looking at <a href="https://arxiv.org/pdf/2208.11970.pdf" rel="nofollow noreferrer">this</a> paper (Page 21.), we can write classifier guidance as: <span class="math-container">$$\nabla_{x}\log p\left(x_{t}\mid y\right) =\nabla_{x}\log\left(\frac{p\left(x_{t}\right)\cdot p\left(y\mid x_{t}\right)}{p\left(y\right)}\right)
      =\nabla_{x}\log p\left(x_{t}\right)+\nabla_{x}\log p\left(y\mid x_{t}\right)-\nabla_{x}\log p\left(y\right)
      =\nabla_{x}\log p\left(x_{t}\right)+\nabla_{x}\log p\left(y\mid x_{t}\right)$$</span>
  and we can amplify the guidance by adding a factor <span class="math-container">$\gamma$</span>:
  <span class="math-container">$$\nabla_{x}\log p\left(x_{t}\mid y\right)=\nabla_{x}\log p\left(x_{t}\right)+\gamma\cdot\nabla_{x}\log p\left(y\mid x_{t}\right)$$</span></p>
  <p>Then to derive classifier-free guidance all we do is rewrite the first equation we saw:
  <span class="math-container">$$\nabla_{x}\log p\left(y\mid x_{t}\right)=\nabla_{x}\log p\left(x_{t}\mid y\right)-\nabla_{x}\log p\left(x_{t}\right)$$</span>
  and substitue it into the 2nd equation we saw to get <span class="math-container">$$\nabla_{x}\log p\left(x_{t}\mid y\right)=\nabla_{x}\log p\left(x_{t}\right)+\gamma\cdot\left(\nabla_{x}\log p\left(x_{t}\mid y\right)-\nabla_{x}\log p\left(x_{t}\right)\right)$$</span>
  and this is classifier-free guidance.</p>
  <p>We can also rewrite it to be <span class="math-container">$$\nabla_{x}\log p\left(x_{t}\mid y\right)=\gamma\cdot\nabla_{x}\log p\left(x_{t}\mid y\right)-\left(\gamma-1\right)\cdot\nabla_{x}\log p\left(x_{t}\right)$$</span></p>
  <h2>The Question</h2>
  <hr />
  <p>It seems reasonable to me that as we can amplify and play with the guidance factor, we could also add a factor <span class="math-container">$\beta$</span> to play with how much we go in the direction of <span class="math-container">$\nabla_{x}\log p\left(x_{t}\mid y\right)$</span> and in the classifier guidance get <span class="math-container">$$\nabla_{x}\log p\left(x_{t}\mid y\right)=\beta\cdot\nabla_{x}\log p\left(x_{t}\right)+\gamma\cdot\nabla_{x}\log p\left(y\mid x_{t}\right)$$</span> and in classifier-free guidance get effectively free control over both terms: <span class="math-container">$$\nabla_{x}\log p\left(x_{t}\mid y\right)=\gamma_{1}\cdot\nabla_{x}\log p\left(x_{t}\mid y\right)-\gamma_{2}\cdot\nabla_{x}\log p\left(x_{t}\right)$$</span></p>
  <p>I wanted to know why adding this extra control over the direction we move in doesn't help, why does anything we choose besides <span class="math-container">$\gamma_{1}=\gamma_{2}+1$</span> yield bad results? Adding this extra factor <span class="math-container">$\beta/\gamma_{1}$</span> seems as intuitive and arbitrary as adding the <span class="math-container">$\gamma$</span> factor at the derivation of Classifier Guidance (at the beginning).</p>
  <p>Is there a deep explanation or at least intuition behind why shouldn't this work?</p>
  
Tags: <generative-model><image-generation><diffusion-models>
Title: Justification of Scaling in Classifier-Free Guidance in Diffusion Models
UserDisplayName: snatchysquid
ThreadText: # Justification of Scaling in Classifier-Free Guidance in Diffusion Models
  
  (Asked by: username_0 on 2023-08-11)
  
  ## Background | Classifier-Free Guidance Derivation
  
  ---
  
  To summarize the derivation of Classifier-Free Guidance, looking at [this](https://arxiv.org/pdf/2208.11970.pdf) paper (Page 21.), we can write classifier guidance as: $$\nabla\_{x}\log p\left(x\_{t}\mid y\right) =\nabla\_{x}\log\left(\frac{p\left(x\_{t}\right)\cdot p\left(y\mid x\_{t}\right)}{p\left(y\right)}\right)
  =\nabla\_{x}\log p\left(x\_{t}\right)+\nabla\_{x}\log p\left(y\mid x\_{t}\right)-\nabla\_{x}\log p\left(y\right)
  =\nabla\_{x}\log p\left(x\_{t}\right)+\nabla\_{x}\log p\left(y\mid x\_{t}\right)$$
  and we can amplify the guidance by adding a factor $\gamma$:
  $$\nabla\_{x}\log p\left(x\_{t}\mid y\right)=\nabla\_{x}\log p\left(x\_{t}\right)+\gamma\cdot\nabla\_{x}\log p\left(y\mid x\_{t}\right)$$
  
  Then to derive classifier-free guidance all we do is rewrite the first equation we saw:
  $$\nabla\_{x}\log p\left(y\mid x\_{t}\right)=\nabla\_{x}\log p\left(x\_{t}\mid y\right)-\nabla\_{x}\log p\left(x\_{t}\right)$$
  and substitue it into the 2nd equation we saw to get $$\nabla\_{x}\log p\left(x\_{t}\mid y\right)=\nabla\_{x}\log p\left(x\_{t}\right)+\gamma\cdot\left(\nabla\_{x}\log p\left(x\_{t}\mid y\right)-\nabla\_{x}\log p\left(x\_{t}\right)\right)$$
  and this is classifier-free guidance.
  
  We can also rewrite it to be $$\nabla\_{x}\log p\left(x\_{t}\mid y\right)=\gamma\cdot\nabla\_{x}\log p\left(x\_{t}\mid y\right)-\left(\gamma-1\right)\cdot\nabla\_{x}\log p\left(x\_{t}\right)$$
  
  ## The Question
  
  ---
  
  It seems reasonable to me that as we can amplify and play with the guidance factor, we could also add a factor $\beta$ to play with how much we go in the direction of $\nabla\_{x}\log p\left(x\_{t}\mid y\right)$ and in the classifier guidance get $$\nabla\_{x}\log p\left(x\_{t}\mid y\right)=\beta\cdot\nabla\_{x}\log p\left(x\_{t}\right)+\gamma\cdot\nabla\_{x}\log p\left(y\mid x\_{t}\right)$$ and in classifier-free guidance get effectively free control over both terms: $$\nabla\_{x}\log p\left(x\_{t}\mid y\right)=\gamma\_{1}\cdot\nabla\_{x}\log p\left(x\_{t}\mid y\right)-\gamma\_{2}\cdot\nabla\_{x}\log p\left(x\_{t}\right)$$
  
  I wanted to know why adding this extra control over the direction we move in doesn't help, why does anything we choose besides $\gamma\_{1}=\gamma\_{2}+1$ yield bad results? Adding this extra factor $\beta/\gamma\_{1}$ seems as intuitive and arbitrary as adding the $\gamma$ factor at the derivation of Classifier Guidance (at the beginning).
  
  Is there a deep explanation or at least intuition behind why shouldn't this work?
  
  **Tags:** generative-model, image-generation, diffusion-models
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2024-11-08. Score: 0
  
  There is an intuitive reason if you look past the logarithm and the score.
  
  The fact is that classifier-free guidance is really a geometric average between the conditional and unconditional probabilities. Basically,
  
  $$p(x \mid y) \propto p(x \mid y)^{\gamma}p(x)^{1 - \gamma}$$
  
  Why is this valid? In the limiting case, consider a scenario where $x$ is actually independent of $y$, i.e. $p(x \mid y) = p(x)$. In that case,
  
  $$
  p(x \mid y) \propto p(x \mid y)^{\gamma}p(x)^{1 - \gamma} = p(x)^\gamma p(x)^{1 - \gamma} \propto p(x)
  $$
  
  It is consistent.
  
  However, if you consider arbitrary scaling factors $\gamma\_1$ and $-\gamma\_2$, you lose the geometric consistency. Effectively, what is going on is:
  
  $$
  p(x \mid y) ^{\gamma\_1}p(x)^{-\gamma\_2} \propto p(x \mid y)^{\tau}
  $$
  
  Where $\tau$ is some constant. The limiting case where $x$ and $y$ are independent makes this obvious, you get $p(x)^{\gamma\_1 - \gamma\_2}$. So as long as $\gamma\_2 \neq \gamma\_1 - 1$, you are sampling from some $p(x \mid y)^\tau$ (which is usually strange stuff way off the data manifold), while you really want to just sample from $p(x \mid y)^1 = p(x \mid y)$.
  
  
  ---
  
  ### Answer by username_2 on 2023-09-10. Score: 0
  
  **You probably *could* express the guidance score in that form, but it's important understand the intention behind how things are presented.**
  
  In this context, we only really care about the *relative* magnitude of the conditional and unconditional terms of the guidance score. When controlling the generation of images with class conditioning, it wouldn't matter how large the conditional term is in isolation if it's overshadowed by unconditional term.
  
  By writing the guidance this way, we think about the score in terms of which signal (either conditional or unconditional) is *dominating* the overall feedback to the model. This "level of domination" only needs one knob to express ($\gamma$).
  
  If you were writing this paper, you *could* add an extra scaling term around the guidance score, adjusting the *overall* magnitude of the feedback and achieve the same degrees of freedom as your suggested formulation. Looking at the [original classifier-free guidance paper](https://arxiv.org/pdf/2207.12598.pdf), it seems like there is a similar hyperparameter that you could adjust, although it's not exactly the same.
  
  However, you could just as reasonably omit this term as (1) it would require extra space to explain, but it (2) doesn't change the formula significantly, and (3) is an adjustment that readers could reasonably arrive to themselves, if they *do* end up wanting this extra flexibility.
  
  > **Comments:**
  > **username_0**: First of all, I must again state that, from my experiments using huggingface models, using any parameter other than $\gamma_{1} = \gamma_{2} + 1$ yields awful results (not even a normal picture), so I'm not sure your answer is correct. Secondly, what extra scaling hyperparameter are you talking about in the original CFG paper?  
  > **username_2**: @snatchysquid What values were you trying that worked/didn't work? Can you also link to the models that you were using/where you were making the changes?  
  > **username_0**: I used for example the model in the following colab [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers_doc/en/pytorch/write_own_pipeline.ipynb) and simply change the Classifier-Free Guidance scale. choosing even a difference of 1.25 (instead of 1) between $\gamma_{1},\gamma_{2}$ resulted in either a blank (brownish) image, or a blank image with a small figure that sometimes matches the prompt (e.g. a banana riding on a skateboard and wearing a hat)  
  
  ---
  
  