Answers.Body: <p>I agree with <a href="https://ai.stackexchange.com/a/41564/52501">@Neil</a> answer, as I also strongly believe that cGANs are the actual answer of your problem.</p>
  <p>However, as he suggested, maybe it's worth mentioning that also <a href="https://arxiv.org/pdf/2101.05278.pdf" rel="nofollow noreferrer">GANs inversion</a> can be used to do achieve such results, and it's used when training a new GAN from scratch it's too expensive.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-31T12:44:03.567Z
Answers.LastActivityDate: 2023-07-31T12:44:03.567Z
Answers.UserDisplayName: Alberto
Answers.Body: <p>This is a common use case for GANs, you want the output to be conditioned on some controlled input, as opposed to just random seed data.</p>
  <p>This Medium article <a href="https://towardsdatascience.com/cgan-conditional-generative-adversarial-network-how-to-gain-control-over-gan-outputs-b30620bd0cc8" rel="nofollow noreferrer">cGAN: Conditional Generative Adversarial Network — How to Gain Control Over GAN Outputs</a> walks through creating your example almost exactly (using MNIST digits and selecting which one you want).</p>
  <p>The basic changes you need to make to turn your &quot;freely choose class&quot; GAN to a conditional GAN:</p>
  <ul>
  <li><p>Add input to the generator, <em>in addition to</em> the random seed, that describes the class you want to generate. For a digit generator, that might be a one-hot array for the digit class you want it to make.</p>
  </li>
  <li><p>Add the same input to the discriminator, alongside the image input. That may involve having inputs at different layers, so you can combine CNN and fully-connect layers more easily. Typically you would concatenate the class choice to the flattened last CNN layer, and use this concatenated vector as input to the first fully-connected layer. But you could concat the class data to any nn layer before the output.</p>
  </li>
  <li><p>Train as before, whilst tracking which class is being faked (or real) and ensuring the generator and discriminator are fed the correct class details during training.</p>
  </li>
  </ul>
  <p>You could optionally provide incorrect labels for some real or fake images, and score appropriately, but that is not 100% necessary. The discriminator should detect that something declared to be a '1' looks more like a '2' and mark it as fake, without needing to be specifically trained for it.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-31T12:36:23.087Z
Answers.Text: maybe also GAN inversion could be used, but i agree with you that cGANS are actually the correct answer (+1)
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-31T12:38:21.387Z
Answers.Text: @AlbertoSinigaglia Yes GAN inversion can be a way to adapt an existing trained GAN, or extract more complex descriptive vectors. It's maybe worth an answer as an alternative approach.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-31T12:44:13.503Z
Answers.Text: Yup you're right, done
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-01T04:43:08.347Z
Answers.Text: @NeilSlater If the problem is non-categorical (say, the labels are arbitrary numbers), would a cGAN still be effective at solving the problem? As it seems that cGANs are mostly optimized for categorical data?
Answers.UserDisplayName: JS4137
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-01T06:46:52.163Z
Answers.Text: @JS4137  a cGAN would not learn to generalise to a space like arbitrary numbers. It is not just the very large space, but also the very specific results expected for the smallest differences. Other continuous spaces might still be possible, e.g. what direction the portrait of a person was looking in, or the colour of a generated car's paint. The results might not be as precise as e.g. generating from a ray tracing system, but could be made acceptable using cGAN-like setup
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-01T09:38:06.800Z
Answers.Text: @NeilSlater Sorry to ask a follow-up question, but given a non-categorical but structured multi-input problem (such as given the labels of a car's color, year it was produced, dimensions, and other info about that car, asking the cGAN to produce an image of that car), would a cGAN still be workable? In my experimentation it has not worked so well.
Answers.UserDisplayName: JS4137
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-01T16:38:50.440Z
Answers.Text: @JS4137 Yes that's a bit much for cGAN. You are probably getting into something more like CLIP-guided generations - https://wandb.ai/geekyrakshit/stylegan-nada/reports/Digging-Into-StyleGAN-NADA-for-CLIP-Guided-Domain-Adaptation--VmlldzoyMjA5MDU1 - although that won't specifically follow all the technical details you are listing, there may be a similar architecture that could.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-31T12:15:47.387Z
Answers.LastActivityDate: 2023-07-31T12:15:47.387Z
Answers.UserDisplayName: Neil Slater
Body: <p>I am trying to accomplish the reverse of the typical MNIST in machine learning using a GAN - instead of predicting a number from an image of a digit, I want to reconstruct an image of a digit from a number. The traditional GAN, however, isn't designed for this use case, as it is designed to generate images similar to training data directly without being given an input. One way to work around this issue that I've thought of is to take a train feature digit, connect it to a densely-connected layer <code>Dense(784)</code>, reshape it to <code>(28 x 28 x 1)</code>, and then proceed with the generator as one usually does for a GAN. However, this seems like &quot;fooling&quot; the neural network to make up weights out of thin air, and I doubt this would work properly.</p>
  <p>How can I modify a GAN so that it takes single-digit inputs without resorting to the aforementioned approach?</p>
  
Tags: <neural-networks><generative-adversarial-networks><mnist>
Title: Implementing a GAN with control over the output class
UserDisplayName: JS4137
ThreadText: # Implementing a GAN with control over the output class
  
  (Asked by: username_0 on 2023-07-31)
  
  I am trying to accomplish the reverse of the typical MNIST in machine learning using a GAN - instead of predicting a number from an image of a digit, I want to reconstruct an image of a digit from a number. The traditional GAN, however, isn't designed for this use case, as it is designed to generate images similar to training data directly without being given an input. One way to work around this issue that I've thought of is to take a train feature digit, connect it to a densely-connected layer `Dense(784)`, reshape it to `(28 x 28 x 1)`, and then proceed with the generator as one usually does for a GAN. However, this seems like "fooling" the neural network to make up weights out of thin air, and I doubt this would work properly.
  
  How can I modify a GAN so that it takes single-digit inputs without resorting to the aforementioned approach?
  
  **Tags:** neural-networks, generative-adversarial-networks, mnist
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2023-07-31. Score: 4 | Accepted answer
  
  This is a common use case for GANs, you want the output to be conditioned on some controlled input, as opposed to just random seed data.
  
  This Medium article [cGAN: Conditional Generative Adversarial Network — How to Gain Control Over GAN Outputs](https://towardsdatascience.com/cgan-conditional-generative-adversarial-network-how-to-gain-control-over-gan-outputs-b30620bd0cc8) walks through creating your example almost exactly (using MNIST digits and selecting which one you want).
  
  The basic changes you need to make to turn your "freely choose class" GAN to a conditional GAN:
  
  * Add input to the generator, *in addition to* the random seed, that describes the class you want to generate. For a digit generator, that might be a one-hot array for the digit class you want it to make.
  * Add the same input to the discriminator, alongside the image input. That may involve having inputs at different layers, so you can combine CNN and fully-connect layers more easily. Typically you would concatenate the class choice to the flattened last CNN layer, and use this concatenated vector as input to the first fully-connected layer. But you could concat the class data to any nn layer before the output.
  * Train as before, whilst tracking which class is being faked (or real) and ensuring the generator and discriminator are fed the correct class details during training.
  
  You could optionally provide incorrect labels for some real or fake images, and score appropriately, but that is not 100% necessary. The discriminator should detect that something declared to be a '1' looks more like a '2' and mark it as fake, without needing to be specifically trained for it.
  
  > **Comments:**
  > **username_1**: maybe also GAN inversion could be used, but i agree with you that cGANS are actually the correct answer (+1)  
  > **username_2**: @AlbertoSinigaglia Yes GAN inversion can be a way to adapt an existing trained GAN, or extract more complex descriptive vectors. It's maybe worth an answer as an alternative approach.  
  > **username_1**: Yup you're right, done  
  > **username_0**: @username_2 If the problem is non-categorical (say, the labels are arbitrary numbers), would a cGAN still be effective at solving the problem? As it seems that cGANs are mostly optimized for categorical data?  
  > **username_2**: @username_0  a cGAN would not learn to generalise to a space like arbitrary numbers. It is not just the very large space, but also the very specific results expected for the smallest differences. Other continuous spaces might still be possible, e.g. what direction the portrait of a person was looking in, or the colour of a generated car's paint. The results might not be as precise as e.g. generating from a ray tracing system, but could be made acceptable using cGAN-like setup  
  > **username_0**: @username_2 Sorry to ask a follow-up question, but given a non-categorical but structured multi-input problem (such as given the labels of a car's color, year it was produced, dimensions, and other info about that car, asking the cGAN to produce an image of that car), would a cGAN still be workable? In my experimentation it has not worked so well.  
  > **username_2**: @username_0 Yes that's a bit much for cGAN. You are probably getting into something more like CLIP-guided generations - https://wandb.ai/geekyrakshit/stylegan-nada/reports/Digging-Into-StyleGAN-NADA-for-CLIP-Guided-Domain-Adaptation--VmlldzoyMjA5MDU1 - although that won't specifically follow all the technical details you are listing, there may be a similar architecture that could.  
  
  ---
  
  ### Answer by username_1 on 2023-07-31. Score: 3
  
  I agree with [@username_2](https://ai.stackexchange.com/a/41564/52501) answer, as I also strongly believe that cGANs are the actual answer of your problem.
  
  However, as he suggested, maybe it's worth mentioning that also [GANs inversion](https://arxiv.org/pdf/2101.05278.pdf) can be used to do achieve such results, and it's used when training a new GAN from scratch it's too expensive.
  
  
  ---
  
  