Body: <p>I am mapping rather complex data into what essentially amounts to a greyscale image to take better advantage of GANs for generative means. Here is an example of some real data:
  <a href="https://i.sstatic.net/RKSNc.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/RKSNc.png" alt="Real data example of image" /></a></p>
  <p>All real data is of the same shape (108 across x 12 high), with an enforced border region of two pixels along each edge - so the actual data is captured by the central 104x8 pixel data. There is significant coupling between individual pixels, so I applied a CNN in the network to try and learn these dependencies, with unequal kernels because the image is rectangular. A rough description for each row of pixels is as follows (note that each row is normalised between 0 and 1 for the network, with 0 corresponding to border region):</p>
  <ol>
  <li>Integer values -1 -1 -1 -2 -1 -1 -1 -2... etc.</li>
  <li>Very small float values (~0.03) with a break in the middle (~0.7)</li>
  <li>Integer values -2 -2 -3 -3 -2 -2 -3 -3...</li>
  <li>Float values</li>
  <li>Integer values -3 -4 -4 -4 -3 -4 -4 -4 -3...</li>
  <li>Float values</li>
  <li>Integer values (one hot of 24, moving in blocks of 4)</li>
  <li>Integer values (one hot of 4, repeating in blocks of 4)</li>
  </ol>
  <p>I have applied the WGAN network with gradient penalty implemented in PyTorch. The general generator network has this architecture:</p>
  <ol>
  <li>ConvTranspose2d</li>
  <li>LeakyReLU(0.2)</li>
  <li>BatchNorm2d</li>
  </ol>
  <p>With a sigmoid at the end. The critic looks like:</p>
  <ol>
  <li>Conv2d</li>
  <li>InstanceNorm2d</li>
  <li>LeakyReLU(0.2)</li>
  </ol>
  <p>With no activation at the end of the critic. I have a dataset size of 1795, sadly I don't really have any conventional means to increase this. I train the critic for 5 rounds for each round of generator training. After 900 epochs, an example of the generated output is as follows:</p>
  <p><a href="https://i.sstatic.net/WJfrh.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/WJfrh.png" alt="Fake data with WGAN-GP" /></a></p>
  <p>So looks quite close, but converting this back into the actual form of my data results in some weird results because of the variation in pixels along the float rows.
  There are also some weird bits on the side (it doesn't seem to have properly learnt the border). The first few pixels in the rows are also unusual in the real data, with the first few for some of the rows basically being a border region - this behaviour seems to have been learnt for some rows (e.g. row 5), but not others. This may be because of my unequal kernels, I'm not sure.</p>
  <p>Some current improvements I'm considering are to add gaussian noise to the data (this may help as it removes the integer aspect of some pixels), augment the training set both with noise and with conventional methods such as rotating the image 180 degrees, adding extra border regions and translating the images etc.. If anyone has any insight into how I can further improve output I would really appreciate it. This image is actually a simplification of the dataset, and eventually, I want to have less regular sequencing in the pixels. Thanks!</p>
  <p><em>(Edit)</em>
  Here is an image of the critic (D) vs generator (G) loss over the 900 epochs used to generate the fake data:</p>
  <p><a href="https://i.sstatic.net/zWQ7Q.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/zWQ7Q.png" alt="loss landscape of critic vs generator" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-02-25T12:32:12.657Z
Comments.Text: do you have some plots of the losses? it would be helpful to know at the very least if the model already converged or not (gans can take days to converge even for small datasets like your) and if the generator vs discriminator losses are correlated with each other or behaving randomly. A plot of the gradient penalty would be insightful as well, in my experience sometimes it can blow up to insane values and take over the whole adversarial loss.
Comments.UserDisplayName: Edoardo Guerriero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-02-26T11:01:22.443Z
Comments.Text: Hi, thank you for your comment! I don't have a plot of the GP right now but I will try and recreate one (I didn't know it could blow up like that actually), I've included a plot of the generator and critic losses.
Comments.UserDisplayName: Zintho
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-29T16:09:52.533Z
Comments.Text: The purpose of the generator is to generate images. You don’t add sigmoid to your generator architecture
Comments.UserDisplayName: rajan subramanian
Tags: <convolutional-neural-networks><generative-adversarial-networks><image-generation><wasserstein-gan><dc-gan>
Title: Possible improvements to WGAN-GP output images
UserDisplayName: Zintho
ThreadText: # Possible improvements to WGAN-GP output images
  
  (Asked by: username_0 on 2022-02-25)
  
  I am mapping rather complex data into what essentially amounts to a greyscale image to take better advantage of GANs for generative means. Here is an example of some real data:
  [![Real data example of image](https://i.sstatic.net/RKSNc.png)](https://i.sstatic.net/RKSNc.png)
  
  All real data is of the same shape (108 across x 12 high), with an enforced border region of two pixels along each edge - so the actual data is captured by the central 104x8 pixel data. There is significant coupling between individual pixels, so I applied a CNN in the network to try and learn these dependencies, with unequal kernels because the image is rectangular. A rough description for each row of pixels is as follows (note that each row is normalised between 0 and 1 for the network, with 0 corresponding to border region):
  
  1. Integer values -1 -1 -1 -2 -1 -1 -1 -2... etc.
  2. Very small float values (~0.03) with a break in the middle (~0.7)
  3. Integer values -2 -2 -3 -3 -2 -2 -3 -3...
  4. Float values
  5. Integer values -3 -4 -4 -4 -3 -4 -4 -4 -3...
  6. Float values
  7. Integer values (one hot of 24, moving in blocks of 4)
  8. Integer values (one hot of 4, repeating in blocks of 4)
  
  I have applied the WGAN network with gradient penalty implemented in PyTorch. The general generator network has this architecture:
  
  1. ConvTranspose2d
  2. LeakyReLU(0.2)
  3. BatchNorm2d
  
  With a sigmoid at the end. The critic looks like:
  
  1. Conv2d
  2. InstanceNorm2d
  3. LeakyReLU(0.2)
  
  With no activation at the end of the critic. I have a dataset size of 1795, sadly I don't really have any conventional means to increase this. I train the critic for 5 rounds for each round of generator training. After 900 epochs, an example of the generated output is as follows:
  
  [![Fake data with WGAN-GP](https://i.sstatic.net/WJfrh.png)](https://i.sstatic.net/WJfrh.png)
  
  So looks quite close, but converting this back into the actual form of my data results in some weird results because of the variation in pixels along the float rows.
  There are also some weird bits on the side (it doesn't seem to have properly learnt the border). The first few pixels in the rows are also unusual in the real data, with the first few for some of the rows basically being a border region - this behaviour seems to have been learnt for some rows (e.g. row 5), but not others. This may be because of my unequal kernels, I'm not sure.
  
  Some current improvements I'm considering are to add gaussian noise to the data (this may help as it removes the integer aspect of some pixels), augment the training set both with noise and with conventional methods such as rotating the image 180 degrees, adding extra border regions and translating the images etc.. If anyone has any insight into how I can further improve output I would really appreciate it. This image is actually a simplification of the dataset, and eventually, I want to have less regular sequencing in the pixels. Thanks!
  
  *(Edit)*
  Here is an image of the critic (D) vs generator (G) loss over the 900 epochs used to generate the fake data:
  
  [![loss landscape of critic vs generator](https://i.sstatic.net/zWQ7Q.png)](https://i.sstatic.net/zWQ7Q.png)
  
  **Tags:** convolutional-neural-networks, generative-adversarial-networks, image-generation, wasserstein-gan, dc-gan
  > **Comments:**
  > **username_1**: do you have some plots of the losses? it would be helpful to know at the very least if the model already converged or not (gans can take days to converge even for small datasets like your) and if the generator vs discriminator losses are correlated with each other or behaving randomly. A plot of the gradient penalty would be insightful as well, in my experience sometimes it can blow up to insane values and take over the whole adversarial loss.  
  > **username_0**: Hi, thank you for your comment! I don't have a plot of the GP right now but I will try and recreate one (I didn't know it could blow up like that actually), I've included a plot of the generator and critic losses.  
  > **username_2**: The purpose of the generator is to generate images. You don’t add sigmoid to your generator architecture  
  
  ---
  
  ## 0 answers
  
  