Answers.Body: <p>I took a look at your <a href="https://pastebin.com/ehMDP7k6" rel="nofollow noreferrer">model</a>. It seems you have incorrect architecture. The <code>Conv2D</code> layers in your D should have following params: <code>(n_filters, kernel=3, padding='same')</code>, where <code>n_filters</code> is the number of filters and it usually should be doubled as per <a href="https://arxiv.org/abs/1511.06434" rel="nofollow noreferrer">DCGAN</a> architecture. You can also use <code>strides</code> but since your images are small it won't make any sense.</p>
  <p>The D network also should not include <code>UpSampling2D</code> layer. This layer can be used in G network instead of <code>Conv2DTranspose</code>.</p>
  <p>Your images should be normalized to the range of <code>[-1, 1]</code> and the activation function of G should be <code>tanh</code>.</p>
  <p>You also included <code>m.add(keras.layers.Reshape((8, 8, 3)))</code> to your G network which is the final size of your data. Since you have the final size in the first layers you don't have to include upsampling <code>Conv2DTranspose</code> layers.</p>
  <p>Your final model should look like</p>
  <pre><code>def make_discriminator_model():
      model = tf.keras.Sequential()
      model.add(layers.Conv2D(8, kernel=3, strides=1, padding='same',
                                       input_shape=[8, 8, 1]))
      model.add(layers.LeakyReLU())
      model.add(layers.Dropout(0.3))
  
      model.add(layers.Conv2D(16, kernel=3, strides=1, padding='same'))
      model.add(layers.LeakyReLU())
      model.add(layers.Dropout(0.3))
  
      model.add(layers.Flatten())
      model.add(layers.Dense(1))
  
      return model
  
  def make_generator_model():
      model = tf.keras.Sequential()
      model.add(layers.Dense(8*8*16, use_bias=False, input_shape=(100,)))
      model.add(layers.BatchNormalization())
      model.add(layers.LeakyReLU())
  
      model.add(layers.Reshape((8, 8, 16)))
  
      model.add(layers.Conv2D(8, kernel=3, strides=1, padding='same'))
      model.add(layers.BatchNormalization())
      model.add(layers.LeakyReLU())
  
      model.add(layers.Conv2D(1, kernel=3, strides=1, padding='same', activation='tanh'))
  
      return model
  </code></pre>
  <p>Note that I did not test the model, so you should adapt it to the shape of your data.</p>
  <p>I encourage you to follow <a href="https://www.tensorflow.org/tutorials/generative/dcgan" rel="nofollow noreferrer">this tutorial</a>.</p>
  <p><strong>Update</strong> regarding overfitting</p>
  <p>GANs do not have overfitting problem in the classical sense. Instead, vanilla GAN have other problems like vanishing gradient. It happens when the D becomes overconfident regarding fake samples. In that case, D stops providing useful information to the G, since its error becomes 0. To find out if training fell into this problem, you should plot D and G loss. It will look as follows:</p>
  <p><a href="https://i.sstatic.net/6sof5.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/6sof5.png" alt="DCGAN loss" /></a></p>
  <p>Another problem is the mode collapse. In that case, the G tricks the D by producing only one type of samples which looks realistic but G won't represent the real data distribution. Therefore, the generated samples will be homogeneous.</p>
  <p>For more information, see <a href="https://arxiv.org/abs/1606.03498" rel="nofollow noreferrer">Improved Techniques for Training GANs</a> and <a href="https://arxiv.org/abs/1701.07875" rel="nofollow noreferrer">Wasserstein GAN</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-17T14:38:06.867Z
Answers.Text: Thank you for the guidance. I certainly built my models with baseless knowledge, and most importantly, I forgot to normalise my image array before feeding into the discriminator, which would explain a lot at my initial problem. I accepted this answer for raising the normalisation problem, and upvoted for the extra guidance.
Answers.UserDisplayName: dee cue
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-17T15:43:20.823Z
Answers.Text: I forgot to mention the overfitting issue. I updated the answer accordingly
Answers.UserDisplayName: Aray Karjauv
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-17T12:37:24.407Z
Answers.LastActivityDate: 2020-11-17T15:42:02.333Z
Answers.LastEditDate: 2020-11-17T15:42:02.333Z
Answers.UserDisplayName: Aray Karjauv
Body: <p>I have a dataset of 3000 8x8 images, and I would like to train a GAN for an image generation purpose.</p>
  <p>I am planning to start with a simple GAN model and see if it overfits. Before training, I try to do a comparison of the discriminator model prediction using real image input against the whole GAN model prediction using random seed input. My thought process is that since this model is not trained yet, the output for real images and fake images by the discriminator should not be predictable.</p>
  <p>However, the discriminator model prediction using real image input always returns a value very close to <code>1.0</code>, and the whole GAN model prediction using random seed input always returns a value near <code>0.5</code> with a small deviation. I suspect that during training, the model would simply pull the <code>0.5</code> value near <code>0.0</code> and would never actually learn from the dataset.</p>
  <p>I try to increase the training parameters and different initializers, but the output is still the same.</p>
  <p>By ruling out the possibility a bad dataset, what could be the reason for this situation?</p>
  <p>This is some sneak peek of the generator and discriminator model building: <a href="https://pastebin.com/ehMDP7k6" rel="nofollow noreferrer">https://pastebin.com/ehMDP7k6</a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-16T19:29:54.467Z
Comments.Text: what do you mean by "a comparison of the discriminator model prediction ... against the whole GAN model prediction"? do you mean the output of your generator for fake and real images?
Comments.UserDisplayName: Aray Karjauv
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-17T11:18:34.023Z
Comments.Text: In other words, I feed real images array into the discriminator model, and it outputs values close to `1.0`. Next, I feed fake images (the one that is generated by the generator) into the discriminator model, and it outputs values close to `0.5`. These happens too often, and it is very predictable.
Comments.UserDisplayName: dee cue
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-17T11:29:21.977Z
Comments.Text: Did you try to generate fake images? Since the main goal of GANs to produce synthetic samples, the main matric will be the quality of generated images. Take a look at [this answer](https://ai.stackexchange.com/questions/18319/gans-should-generator-update-weights-when-discriminator-says-false-continuously/18330#18330). There is a plot of G loss and D loss of DCGAN. We can see that D  loss is near 0, whereas G loss has oscillations, but G was able to produce good samples.
Comments.UserDisplayName: Aray Karjauv
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-17T11:54:34.110Z
Comments.Text: Yes, and it only generates noises. Although, I do notice some preference of a certain pallete, and I think this is a good start. The D accuracy went constant at 0.5 after 20 epochs, and the G accuracy went up to 1.0, and both losses gradually went down. All metrics did oscillate in the middle of the training but in small amount of deviation. Did my model overfit at epoch 20?
Comments.UserDisplayName: dee cue
Tags: <generative-adversarial-networks>
Title: GAN model predictions before training is predictable
UserDisplayName: dee cue
ThreadText: # GAN model predictions before training is predictable
  
  (Asked by: username_0 on 2020-11-16)
  
  I have a dataset of 3000 8x8 images, and I would like to train a GAN for an image generation purpose.
  
  I am planning to start with a simple GAN model and see if it overfits. Before training, I try to do a comparison of the discriminator model prediction using real image input against the whole GAN model prediction using random seed input. My thought process is that since this model is not trained yet, the output for real images and fake images by the discriminator should not be predictable.
  
  However, the discriminator model prediction using real image input always returns a value very close to `1.0`, and the whole GAN model prediction using random seed input always returns a value near `0.5` with a small deviation. I suspect that during training, the model would simply pull the `0.5` value near `0.0` and would never actually learn from the dataset.
  
  I try to increase the training parameters and different initializers, but the output is still the same.
  
  By ruling out the possibility a bad dataset, what could be the reason for this situation?
  
  This is some sneak peek of the generator and discriminator model building: <https://pastebin.com/ehMDP7k6>
  
  **Tags:** generative-adversarial-networks
  
  > **Comments:**
  > **username_1**: what do you mean by "a comparison of the discriminator model prediction ... against the whole GAN model prediction"? do you mean the output of your generator for fake and real images?  
  > **username_0**: In other words, I feed real images array into the discriminator model, and it outputs values close to `1.0`. Next, I feed fake images (the one that is generated by the generator) into the discriminator model, and it outputs values close to `0.5`. These happens too often, and it is very predictable.  
  > **username_1**: Did you try to generate fake images? Since the main goal of GANs to produce synthetic samples, the main matric will be the quality of generated images. Take a look at [this answer](https://ai.stackexchange.com/questions/18319/gans-should-generator-update-weights-when-discriminator-says-false-continuously/18330#18330). There is a plot of G loss and D loss of DCGAN. We can see that D  loss is near 0, whereas G loss has oscillations, but G was able to produce good samples.  
  > **username_0**: Yes, and it only generates noises. Although, I do notice some preference of a certain pallete, and I think this is a good start. The D accuracy went constant at 0.5 after 20 epochs, and the G accuracy went up to 1.0, and both losses gradually went down. All metrics did oscillate in the middle of the training but in small amount of deviation. Did my model overfit at epoch 20?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-11-17. Score: 3 | Accepted answer
  
  I took a look at your [model](https://pastebin.com/ehMDP7k6). It seems you have incorrect architecture. The `Conv2D` layers in your D should have following params: `(n_filters, kernel=3, padding='same')`, where `n_filters` is the number of filters and it usually should be doubled as per [DCGAN](https://arxiv.org/abs/1511.06434) architecture. You can also use `strides` but since your images are small it won't make any sense.
  
  The D network also should not include `UpSampling2D` layer. This layer can be used in G network instead of `Conv2DTranspose`.
  
  Your images should be normalized to the range of `[-1, 1]` and the activation function of G should be `tanh`.
  
  You also included `m.add(keras.layers.Reshape((8, 8, 3)))` to your G network which is the final size of your data. Since you have the final size in the first layers you don't have to include upsampling `Conv2DTranspose` layers.
  
  Your final model should look like
  
  ```
  def make_discriminator_model():
      model = tf.keras.Sequential()
      model.add(layers.Conv2D(8, kernel=3, strides=1, padding='same',
                                       input_shape=[8, 8, 1]))
      model.add(layers.LeakyReLU())
      model.add(layers.Dropout(0.3))
  
      model.add(layers.Conv2D(16, kernel=3, strides=1, padding='same'))
      model.add(layers.LeakyReLU())
      model.add(layers.Dropout(0.3))
  
      model.add(layers.Flatten())
      model.add(layers.Dense(1))
  
      return model
  
  def make_generator_model():
      model = tf.keras.Sequential()
      model.add(layers.Dense(8*8*16, use_bias=False, input_shape=(100,)))
      model.add(layers.BatchNormalization())
      model.add(layers.LeakyReLU())
  
      model.add(layers.Reshape((8, 8, 16)))
  
      model.add(layers.Conv2D(8, kernel=3, strides=1, padding='same'))
      model.add(layers.BatchNormalization())
      model.add(layers.LeakyReLU())
  
      model.add(layers.Conv2D(1, kernel=3, strides=1, padding='same', activation='tanh'))
  
      return model
  
  ```
  
  Note that I did not test the model, so you should adapt it to the shape of your data.
  
  I encourage you to follow [this tutorial](https://www.tensorflow.org/tutorials/generative/dcgan).
  
  **Update** regarding overfitting
  
  GANs do not have overfitting problem in the classical sense. Instead, vanilla GAN have other problems like vanishing gradient. It happens when the D becomes overconfident regarding fake samples. In that case, D stops providing useful information to the G, since its error becomes 0. To find out if training fell into this problem, you should plot D and G loss. It will look as follows:
  
  [![DCGAN loss](https://i.sstatic.net/6sof5.png)](https://i.sstatic.net/6sof5.png)
  
  Another problem is the mode collapse. In that case, the G tricks the D by producing only one type of samples which looks realistic but G won't represent the real data distribution. Therefore, the generated samples will be homogeneous.
  
  For more information, see [Improved Techniques for Training GANs](https://arxiv.org/abs/1606.03498) and [Wasserstein GAN](https://arxiv.org/abs/1701.07875).
  
  > **Comments:**
  > **username_0**: Thank you for the guidance. I certainly built my models with baseless knowledge, and most importantly, I forgot to normalise my image array before feeding into the discriminator, which would explain a lot at my initial problem. I accepted this answer for raising the normalisation problem, and upvoted for the extra guidance.  
  > **username_1**: I forgot to mention the overfitting issue. I updated the answer accordingly  
  
  ---
  
  