Answers.Body: <p>I doubt that you can condition training on a scoring function, if such scoring is not used during training.</p>
  <p>Now, assuming your generator is differentiable, it's already been done the &quot;n-step gradient update&quot; on the latent, via gradient descent:
  <span class="math-container">$$
  z_{t+1} = z_{t} + \alpha \nabla_{z_t} D(G(z_t))
  $$</span>
  given <span class="math-container">$D$</span> the discriminator, <span class="math-container">$G$</span> the generator and <span class="math-container">$z$</span> the latent noise vector (might be missing a <span class="math-container">$-$</span> in the gradient update, depending if the discriminator is trained to predict fakes or reals as <span class="math-container">$1$</span>)</p>
  <p>This approach would be sound if we assume that <span class="math-container">$D$</span> is a perfect discriminator, which is hardly the case in real life, thus using it as a score function is very likely to lead to some samples that will indeed fool the discriminator, but are visually worse for humans</p>
  <p>If <span class="math-container">$D$</span> is not a discriminator, but maybe some other non differentiable scoring function, you can use zero-order optimization to solve estimate the gradient <span class="math-container">$\nabla_{G(z_t)}S(G(z_t))$</span>, or even directly the gradient with respect to <span class="math-container">$z_t$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-06T13:45:37.650Z
Answers.Text: Thanks! It makes sense. 
  
  However for the conditioning method, if assuming we have access to the training, will this be a better method?
Answers.UserDisplayName: terenceflow
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-06T17:09:26.043Z
Answers.Text: @terenceflow not really, the problem is always that either you have a perfect scoring method, or you will have a bad result... if you check for example AlphaFold, you can consider this the same idea, where they had a model to predict a "good initial guess", and then a very good differentiable model that starting from a good guess, creates a good folding
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-06T10:43:21.150Z
Answers.LastActivityDate: 2023-08-06T10:43:21.150Z
Answers.UserDisplayName: Alberto
Answers.Body: <p>For the first suggestion, are you suggesting doing gradient descent with objective <span class="math-container">$-S$</span>? If so (and <span class="math-container">$S$</span> is differentiable), that's definitely possible. I would suggest looking at the <a href="https://arxiv.org/pdf/2003.03808.pdf" rel="nofollow noreferrer">PULSE paper</a>. They do image superresolution over face images by searching through the latent space of a GAN to find high resolution generations that match the given low-resolution image.</p>
  <p>The main thing to keep in mind here is that not all areas in the latent space correspond to realistic generations. You probably want areas that correspond to latents that were likely seen during training (e.g., latents with high probability under the prior). The PULSE paper does this by constraining the search space to a hypersphere centered at the origin. <span class="math-container">$S$</span> here would be measuring how much the downscaled image deviates.</p>
  <p><a href="https://arxiv.org/pdf/1904.03189.pdf" rel="nofollow noreferrer">This paper</a> does something similar. This paper also has an additional perceptual loss term using a VGG model. <span class="math-container">$S$</span> here would be measuring the reconstruction loss using both a simple pixel-wise MSE loss and the perceptual loss.</p>
  <p>Finally, an early way of generating images from text-prompts was <a href="https://towardsdatascience.com/generating-images-from-prompts-using-clip-and-stylegan-1f9ed495ddda" rel="nofollow noreferrer">optimizing the input embeddings for an image generator like StyleGAN using CLIP as a loss function</a>.</p>
  <p>I'm not too sure about your second and third suggestion, though. The second sounds a bit like gradient descent -- how do you know what &quot;the direction of <span class="math-container">$S$</span>&quot; is? The third sounds like <a href="https://arxiv.org/abs/1411.1784" rel="nofollow noreferrer">Conditional GANs</a>, but I'm not aware of work that conditions on the &quot;quality&quot; of the input.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-02T13:53:56.450Z
Answers.LastActivityDate: 2023-08-03T01:25:14.447Z
Answers.LastEditDate: 2023-08-03T01:25:14.447Z
Answers.UserDisplayName: Alexander Wan
Body: <p>Given a generative model, G, trained on a dataset D. This generative model can be either GAN or Diffusion based. Supposed each sample, x_i, generated by G, can be evaluated by a readily available scoring function, S(x_i).</p>
  <p>What are the possible ways to navigate the latent space of G to find generated samples which maximizes S(x_i)?</p>
  <p>Here are some of the possible ways I have thought:</p>
  <ol>
  <li><p>Random samplings from G to form a dataset. For each sample, evaluate it with S, then finetune G to maximise S with loss = -S. Keep repeating until G only produces samples with high S values. This is motivated from RLHF but use backprop instead of RL loss.</p>
  </li>
  <li><p>Fix G, start with a random latent, l_1 to sample x_1 from G. Then slowly find a better x_2 by going into the direction of S, e.g. l_2 = l_1 + alpha * s(x_1) = l_1 + alpha * s( g(l_1)). Is this possible, is it fair to assume P(l, s(1)) is smooth?</p>
  </li>
  <li><p>Condition the training of G with S, i.e G(X|S(X)). In this way, we can specific any desired score as an input to the generative model.</p>
  </li>
  </ol>
  <p>Are any of these ideas possible? Otherwise, are there better or more simpler approaches?</p>
  
Tags: <neural-networks><reinforcement-learning><deep-learning><optimization><generative-model>
Title: Maximize a scoring function within the latent space of a generative model
UserDisplayName: terenceflow
ThreadText: # Maximize a scoring function within the latent space of a generative model
  
  (Asked by: username_0 on 2023-08-02)
  
  Given a generative model, G, trained on a dataset D. This generative model can be either GAN or Diffusion based. Supposed each sample, x\_i, generated by G, can be evaluated by a readily available scoring function, S(x\_i).
  
  What are the possible ways to navigate the latent space of G to find generated samples which maximizes S(x\_i)?
  
  Here are some of the possible ways I have thought:
  
  1. Random samplings from G to form a dataset. For each sample, evaluate it with S, then finetune G to maximise S with loss = -S. Keep repeating until G only produces samples with high S values. This is motivated from RLHF but use backprop instead of RL loss.
  2. Fix G, start with a random latent, l\_1 to sample x\_1 from G. Then slowly find a better x\_2 by going into the direction of S, e.g. l\_2 = l\_1 + alpha \* s(x\_1) = l\_1 + alpha \* s( g(l\_1)). Is this possible, is it fair to assume P(l, s(1)) is smooth?
  3. Condition the training of G with S, i.e G(X|S(X)). In this way, we can specific any desired score as an input to the generative model.
  
  Are any of these ideas possible? Otherwise, are there better or more simpler approaches?
  
  **Tags:** neural-networks, reinforcement-learning, deep-learning, optimization, generative-model
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2023-08-06. Score: 1
  
  I doubt that you can condition training on a scoring function, if such scoring is not used during training.
  
  Now, assuming your generator is differentiable, it's already been done the "n-step gradient update" on the latent, via gradient descent:
  $$
  z\_{t+1} = z\_{t} + \alpha \nabla\_{z\_t} D(G(z\_t))
  $$
  given $D$ the discriminator, $G$ the generator and $z$ the latent noise vector (might be missing a $-$ in the gradient update, depending if the discriminator is trained to predict fakes or reals as $1$)
  
  This approach would be sound if we assume that $D$ is a perfect discriminator, which is hardly the case in real life, thus using it as a score function is very likely to lead to some samples that will indeed fool the discriminator, but are visually worse for humans
  
  If $D$ is not a discriminator, but maybe some other non differentiable scoring function, you can use zero-order optimization to solve estimate the gradient $\nabla\_{G(z\_t)}S(G(z\_t))$, or even directly the gradient with respect to $z\_t$
  
  > **Comments:**
  > **username_0**: Thanks! It makes sense.   However for the conditioning method, if assuming we have access to the training, will this be a better method?  
  > **username_1**: @username_0 not really, the problem is always that either you have a perfect scoring method, or you will have a bad result... if you check for example AlphaFold, you can consider this the same idea, where they had a model to predict a "good initial guess", and then a very good differentiable model that starting from a good guess, creates a good folding  
  
  ---
  
  ### Answer by username_2 on 2023-08-02. Score: 1
  
  For the first suggestion, are you suggesting doing gradient descent with objective $-S$? If so (and $S$ is differentiable), that's definitely possible. I would suggest looking at the [PULSE paper](https://arxiv.org/pdf/2003.03808.pdf). They do image superresolution over face images by searching through the latent space of a GAN to find high resolution generations that match the given low-resolution image.
  
  The main thing to keep in mind here is that not all areas in the latent space correspond to realistic generations. You probably want areas that correspond to latents that were likely seen during training (e.g., latents with high probability under the prior). The PULSE paper does this by constraining the search space to a hypersphere centered at the origin. $S$ here would be measuring how much the downscaled image deviates.
  
  [This paper](https://arxiv.org/pdf/1904.03189.pdf) does something similar. This paper also has an additional perceptual loss term using a VGG model. $S$ here would be measuring the reconstruction loss using both a simple pixel-wise MSE loss and the perceptual loss.
  
  Finally, an early way of generating images from text-prompts was [optimizing the input embeddings for an image generator like StyleGAN using CLIP as a loss function](https://towardsdatascience.com/generating-images-from-prompts-using-clip-and-stylegan-1f9ed495ddda).
  
  I'm not too sure about your second and third suggestion, though. The second sounds a bit like gradient descent -- how do you know what "the direction of $S$" is? The third sounds like [Conditional GANs](https://arxiv.org/abs/1411.1784), but I'm not aware of work that conditions on the "quality" of the input.
  
  
  ---
  
  