Body: <p>I tried to train a ViT-GAN model (from this <a href="https://github.com/YigitGunduc/vit-gan/tree/master" rel="nofollow noreferrer">repo</a>) on my database, where i have images as input and output. The input image is a PNG map of a path planning problem. Red channel is obstacle map, green and blue is one pixel with start/goal. The output image would be the planned path on the red channel and this is what i am trying to teach the model. The model from the repo works nicely on the included examples and that is a much more complex problem than mine, so i naturally thought this network would solve my problem without much further tuning.</p>
  <p>My issue is that even without touching the network, just replacing the training data to mine it completely messes up the output. I tried tweaking the network in so many ways, but the result always remains the same -&gt; either the same structured garbage output for any given input (white patch on black), or some noisy rgb patches on black background. I just cant get it to learn the expected output.</p>
  <p><a href="https://i.sstatic.net/Z4r3M7sm.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Z4r3M7sm.png" alt="This is an example of what i usually get" /></a></p>
  <p>On the left is the given input and on the right is the expected output. Middle is the output of the network. I read about training GAN's on i get they are volatile and hard to find the right structure/parameter set, but i just cant figure out what to do next with it. What i already tried:</p>
  <ul>
  <li>Making the network smaller by removing some transformer/convulutional layers</li>
  <li>Lowering the number of filters hence my problem is not that complex</li>
  <li>Adjusting the learning rates of the generator/discriminator -&gt; together and separately</li>
  <li>Changing the activation function of the output layer of the discriminator to sigmoid as suggested in another similar post</li>
  <li>Removed all ReLU activation functions and replaced them with LeakyReLU to address possible vanishing gradient problems</li>
  <li>Adding Wasserstein loss to the discriminator loss function to avoid model collapse</li>
  <li>Changing mean diff of pixels in the generator loss function to sum diff (to punish full black output more)</li>
  <li>Use bias to avoid vanishing gradient</li>
  <li>Changing the random initializer of filter from (0.0, 0.02) -&gt; (0.0, 1.0) to make the initial filter more varied</li>
  <li>Changing the lambda, batch size, ff_dim, number of heads, patch size, embed dim, projection dim params</li>
  <li>Train for 200 epochs and for 20 also. All for the same output structure.</li>
  <li>Train on smaller portion of dataset (400 images), and on larger (1000 and 3000 images)</li>
  </ul>
  <p>I am aware that this problem could be solved with other (maybe simplier) netwroks also, but i would like to make this GAN work and understand why it wouldn't work in the first place. I added my current state of changes to this <a href="https://github.com/MrCaptainLord/Vit-GAN-example" rel="nofollow noreferrer">repo</a>. I have no idea where to look next. If someone has some suggestions please feel free to share. If you suggest to adjust some params please write a concrete value. If i forgot anything from the description let me know.</p>
  <p>Edit1: Some explanation regarding the included repositories. The train.py and the models.py files contain the logic. In my repository i removed the code from those files, which i dont use, eg. random jitter and random crop. The train.py contains the loss calculation and the learning rates, as well as the prepocessing steps of the data. I think there are no bugs in the code, but if you notice something please let me know. At the preprocessing steps i changed the decode_jpg to decode_png. The models.py file contains the structure of the model, where i commented out some parts of the original layers and removed the duplicate functions. Also i added a LeakyReLU layer in the transformer class and removed the ReLU activation function from the dense layer. Overall i operated with 64x64 RGB images, but i tried with the original 256x256 structure as well and got the same faulty results. In my repo i included 3 sample images from my training set as well. The evaluate.py file contains the metric calculation and image plotting of the results. There i also removed the random jitter and crop funcitons and in the pre_process function i changed the upscaling way from (output + 0.5) x 255 to (output + 1) x 127.5 . These three files can be found in the src folder of the original repository.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-12-08T14:36:44.587Z
Comments.Text: In general, I would suggest you need lot more data/examples. I don't remember doing GAN, but I think dataset size you suggested (3000 examples) is very small. If I were you, I would add lot more examples. Just to make sure what is the dataset size in the repo vs what is your dataset size? Also, make sure you've right dataset. Many a times, there will issues with dataset that no amount of training/tuning can fix!
Comments.UserDisplayName: tired and bored dev
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-12-08T16:53:12.030Z
Comments.Text: I see. I will give it a try. The dataset included in the original repo has 3000 training images and under 100 epochs it learns well. I tried with a 400, 1000 and 3200 sized dataset as well. For the size of 3200 images i tried at the beginning, but i will try again and update here.
Comments.UserDisplayName: Adam Bencsik
Tags: <neural-networks><machine-learning><deep-learning><generative-adversarial-networks><supervised-learning>
Title: How to properly train a GAN model?
UserDisplayName: Adam Bencsik
ThreadText: # How to properly train a GAN model?
  
  (Asked by: username_0 on 2024-12-07)
  
  I tried to train a ViT-GAN model (from this [repo](https://github.com/YigitGunduc/vit-gan/tree/master)) on my database, where i have images as input and output. The input image is a PNG map of a path planning problem. Red channel is obstacle map, green and blue is one pixel with start/goal. The output image would be the planned path on the red channel and this is what i am trying to teach the model. The model from the repo works nicely on the included examples and that is a much more complex problem than mine, so i naturally thought this network would solve my problem without much further tuning.
  
  My issue is that even without touching the network, just replacing the training data to mine it completely messes up the output. I tried tweaking the network in so many ways, but the result always remains the same -> either the same structured garbage output for any given input (white patch on black), or some noisy rgb patches on black background. I just cant get it to learn the expected output.
  
  [![This is an example of what i usually get](https://i.sstatic.net/Z4r3M7sm.png)](https://i.sstatic.net/Z4r3M7sm.png)
  
  On the left is the given input and on the right is the expected output. Middle is the output of the network. I read about training GAN's on i get they are volatile and hard to find the right structure/parameter set, but i just cant figure out what to do next with it. What i already tried:
  
  * Making the network smaller by removing some transformer/convulutional layers
  * Lowering the number of filters hence my problem is not that complex
  * Adjusting the learning rates of the generator/discriminator -> together and separately
  * Changing the activation function of the output layer of the discriminator to sigmoid as suggested in another similar post
  * Removed all ReLU activation functions and replaced them with LeakyReLU to address possible vanishing gradient problems
  * Adding Wasserstein loss to the discriminator loss function to avoid model collapse
  * Changing mean diff of pixels in the generator loss function to sum diff (to punish full black output more)
  * Use bias to avoid vanishing gradient
  * Changing the random initializer of filter from (0.0, 0.02) -> (0.0, 1.0) to make the initial filter more varied
  * Changing the lambda, batch size, ff\_dim, number of heads, patch size, embed dim, projection dim params
  * Train for 200 epochs and for 20 also. All for the same output structure.
  * Train on smaller portion of dataset (400 images), and on larger (1000 and 3000 images)
  
  I am aware that this problem could be solved with other (maybe simplier) netwroks also, but i would like to make this GAN work and understand why it wouldn't work in the first place. I added my current state of changes to this [repo](https://github.com/MrCaptainLord/Vit-GAN-example). I have no idea where to look next. If someone has some suggestions please feel free to share. If you suggest to adjust some params please write a concrete value. If i forgot anything from the description let me know.
  
  Edit1: Some explanation regarding the included repositories. The train.py and the models.py files contain the logic. In my repository i removed the code from those files, which i dont use, eg. random jitter and random crop. The train.py contains the loss calculation and the learning rates, as well as the prepocessing steps of the data. I think there are no bugs in the code, but if you notice something please let me know. At the preprocessing steps i changed the decode\_jpg to decode\_png. The models.py file contains the structure of the model, where i commented out some parts of the original layers and removed the duplicate functions. Also i added a LeakyReLU layer in the transformer class and removed the ReLU activation function from the dense layer. Overall i operated with 64x64 RGB images, but i tried with the original 256x256 structure as well and got the same faulty results. In my repo i included 3 sample images from my training set as well. The evaluate.py file contains the metric calculation and image plotting of the results. There i also removed the random jitter and crop funcitons and in the pre\_process function i changed the upscaling way from (output + 0.5) x 255 to (output + 1) x 127.5 . These three files can be found in the src folder of the original repository.
  
  **Tags:** neural-networks, machine-learning, deep-learning, generative-adversarial-networks, supervised-learning
  > **Comments:**
  > **username_1**: In general, I would suggest you need lot more data/examples. I don't remember doing GAN, but I think dataset size you suggested (3000 examples) is very small. If I were you, I would add lot more examples. Just to make sure what is the dataset size in the repo vs what is your dataset size? Also, make sure you've right dataset. Many a times, there will issues with dataset that no amount of training/tuning can fix!  
  > **username_0**: I see. I will give it a try. The dataset included in the original repo has 3000 training images and under 100 epochs it learns well. I tried with a 400, 1000 and 3200 sized dataset as well. For the size of 3200 images i tried at the beginning, but i will try again and update here.  
  
  ---
  
  ## 0 answers
  
  