Body: <p>I am training GAN on SVHN dataset (house numbers in Google Street View images, dimensions: 3x32x32 - 3 color channels).</p>
  <p>The problem is that it performs worse after some training (e.g. after 50 epochs) than after only 2.
  Could you please check out my code? Maybe you will be able to notice what can I improve.</p>
  <p>I have already tweaked betas in ADAM optimizer (it helped a little bit, because before that, with default settings, d_loss went to 0 after 5 epochs). I also added an extra discriminator training step.</p>
  <p>You can find the code below:</p>
  <pre><code>batch_size=128
  
  # the input data is scaled to a mean of 0.5 and a standard deviation of 0.5:
  transform = transforms.Compose([
                  transforms.ToTensor(),
                  transforms.Normalize(mean=(0.5,), std=(0.5,)),
                  transforms.Lambda(lambda x: x.view(-1))])
  
  
  # wypłaszczanie do loadera (view) albo konwolucyjne
  # lambda
  train_dataset = SVHN(root=&quot;.&quot;, split='train', download=True,
      transform=transform) 
  
  test_dataset = SVHN(root=&quot;.&quot;, split='test', download=True,
      transform=transform) 
  
  train_loader = DataLoader(dataset=train_dataset, batch_size=batch_size, shuffle=True, drop_last=True, num_workers=0)
  test_loader = DataLoader(dataset=test_dataset, batch_size=batch_size, shuffle=True, drop_last=True, num_workers=0)
  
  # Initialize random noise
  
  def noise(size):
      n = torch.randn(size, 100)
      return n.to(device)
  
  # Define the generator model
  
  class Generator(nn.Module):
      def __init__(self):
          super().__init__()
          self.model = nn.Sequential(
                                  # take a 100-dimensional input (random noise)
                                  nn.Linear(100, 256),
                                  nn.LeakyReLU(0.2),
                                  nn.Linear(256, 512),
                                  nn.LeakyReLU(0.2),
                                  nn.Linear(512, 1024),
                                  nn.LeakyReLU(0.2),
                                  nn.Linear(1024, 3*32*32),
                                  nn.Tanh()
  
                              )
  
      def forward(self, x): return self.model(x)
  
  # Define the discriminator model
  
  class Discriminator(nn.Module):
      def __init__(self):
          super().__init__()
          self.model = nn.Sequential( 
                                  nn.Linear(3*32*32, 1024),
                                  nn.LeakyReLU(0.2),
                                  nn.Dropout(0.3),
                                  nn.Linear(1024, 512),
                                  nn.LeakyReLU(0.2),
                                  nn.Dropout(0.3),
                                  nn.Linear(512, 256),
                                  nn.LeakyReLU(0.2),
                                  nn.Dropout(0.3),
                                  nn.Linear(256, 1),
                                  nn.Sigmoid()
  
                              )
      def forward(self, x): return self.model(x)
  
  # define generator training - input is fake data
  def generator_train_step(fake_data):
      # reset the gradient so the parameters will update correctly
      g_optimizer.zero_grad()
  
      # predict the output of the discriminator on fake data 
      prediction_fake = discriminator(fake_data)
  
      # torch.ones, because we want 1s outputted by the discriminator when training the generator
      error = loss(prediction_fake, torch.ones(len(real_data), 1).to(device))
      error.backward()
  
      g_optimizer.step()
      return error
  
  # define discriminator training:
  
  # discriminator as an input takes real data and fake data
  def discriminator_train_step(real_data, fake_data):
      # reset the gradient so the parameters will update correctly
      d_optimizer.zero_grad()
    
      #print(real_data.shape)
      #print(fake_data.shape)
    
      prediction_real = discriminator(real_data)
  
      # calculate loss on real data (expected 1, so torch.ones)
      real_loss = loss(prediction_real, torch.ones(len(real_data),1).to(device))
      real_loss.backward(retain_graph=True)
  
      prediction_fake = discriminator(fake_data)
    
      # calculate loss on fake data (expected 0, so torch.zeros)
      fake_loss = loss(prediction_fake, torch.zeros(len(fake_data), 1).to(device))
      fake_loss.backward(retain_graph=True)
  
      d_optimizer.step()
      return real_loss + fake_loss
  
  lr = 1e-3
  
  discriminator = Discriminator().to(device)
  generator = Generator().to(device)
  d_optimizer = optim.Adam(discriminator.parameters(), lr=lr,  betas=(0.5, 0.999))
  g_optimizer = optim.Adam(generator.parameters(), lr=lr,  betas=(0.4, 0.9))
  loss = nn.BCELoss()
  num_epochs = 50
  log = Report(num_epochs)
  
  for epoch in range(num_epochs):
      N = len(train_loader)
      for i, (images, _) in enumerate(train_loader):
          real_data = images.view(len(images), -1).to(device)
          fake_data = generator(noise(len(real_data))).to(device)
          fake_data = fake_data.detach()
          d_loss = discriminator_train_step(real_data, fake_data)
          fake_data = generator(noise(len(real_data))).to(device)
          
          d_loss = discriminator_train_step(real_data, fake_data)
          
          g_loss = generator_train_step(fake_data)
          
          log.record(epoch+(1+i)/N, d_loss=d_loss.item(), g_loss=g_loss.item(), end='\r')
      log.report_avgs(epoch+1)
      z = torch.randn(10, 100).to(device)
      # 10 pictures of dimensions of 3x32x32:
      sample_images = generator(z).data.cpu().view(10, 3, 32, 32)
      grid = make_grid(sample_images, nrow=4, normalize=True)
      show(grid.cpu().detach().permute(1,2,0), sz=10)
  log.plot_epochs(['d_loss', 'g_loss'])
  </code></pre>
  <p>In case you would like to see the results of training this GAN, I enclose an example generated images after epoch 2:</p>
  <pre><code>EPOCH: 2.000    d_loss: 4.936   g_loss: 1.919   (130.46s - 3131.14s remaining)))
  </code></pre>
  <p><a href="https://i.sstatic.net/qMfmh.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/qMfmh.png" alt="generated images after epoch 2" /></a></p>
  <p>and after epoch 50:
  <a href="https://i.sstatic.net/BETe0.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/BETe0.png" alt="generated images after epoch 50" /></a></p>
  <pre><code>EPOCH: 50.000   d_loss: 1.253   g_loss: 1.038   (3487.47s - 0.00s remaining))
  </code></pre>
  <p>And here is the plot of discriminator loss and generator loss:
  <a href="https://i.sstatic.net/MDJ2Q.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/MDJ2Q.png" alt="enter image description here" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-17T14:53:05.557Z
Comments.Text: Did you normalize your images to [-1, 1]?
Comments.UserDisplayName: Aray Karjauv
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-17T15:36:46.317Z
Comments.Text: @ArayKarjauv oh, I forgot to paste that part of code. But yes, I used transform of transforms.Compose([ transforms.ToTensor(), transforms.Normalize(mean=(0.5,), std=(0.5,)), transforms.Lambda(lambda x: x.view(-1))]) when loading data.
Comments.UserDisplayName: Zu M
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-17T15:48:34.197Z
Comments.Text: It is difficult to say what is wrong with your implementation without debugging it, but GANs are known to be very sensitive to each hyperparameter, which makes the training highly unstable. For example, you don't have any normalization layers. Take a look at this [paper](https://arxiv.org/abs/1606.03498). It contains some tricks that can improve the training. But I would suggest switching to the [Wasserstein GAN](https://arxiv.org/pdf/1704.00028.pdf) or adding $R_1$ [regularization](https://paperswithcode.com/method/r1-regularization)
Comments.UserDisplayName: Aray Karjauv
Tags: <neural-networks><deep-learning><generative-adversarial-networks><pytorch>
Title: GAN performs worse after 50 epochs than after 2
UserDisplayName: Zu M
ThreadText: # GAN performs worse after 50 epochs than after 2
  
  (Asked by: username_0 on 2021-06-17)
  
  I am training GAN on SVHN dataset (house numbers in Google Street View images, dimensions: 3x32x32 - 3 color channels).
  
  The problem is that it performs worse after some training (e.g. after 50 epochs) than after only 2.
  Could you please check out my code? Maybe you will be able to notice what can I improve.
  
  I have already tweaked betas in ADAM optimizer (it helped a little bit, because before that, with default settings, d\_loss went to 0 after 5 epochs). I also added an extra discriminator training step.
  
  You can find the code below:
  
  ```
  batch_size=128
  
  # the input data is scaled to a mean of 0.5 and a standard deviation of 0.5:
  transform = transforms.Compose([
                  transforms.ToTensor(),
                  transforms.Normalize(mean=(0.5,), std=(0.5,)),
                  transforms.Lambda(lambda x: x.view(-1))])
  
  # wypłaszczanie do loadera (view) albo konwolucyjne
  # lambda
  train_dataset = SVHN(root=".", split='train', download=True,
      transform=transform) 
  
  test_dataset = SVHN(root=".", split='test', download=True,
      transform=transform) 
  
  train_loader = DataLoader(dataset=train_dataset, batch_size=batch_size, shuffle=True, drop_last=True, num_workers=0)
  test_loader = DataLoader(dataset=test_dataset, batch_size=batch_size, shuffle=True, drop_last=True, num_workers=0)
  
  # Initialize random noise
  
  def noise(size):
      n = torch.randn(size, 100)
      return n.to(device)
  
  # Define the generator model
  
  class Generator(nn.Module):
      def __init__(self):
          super().__init__()
          self.model = nn.Sequential(
                                  # take a 100-dimensional input (random noise)
                                  nn.Linear(100, 256),
                                  nn.LeakyReLU(0.2),
                                  nn.Linear(256, 512),
                                  nn.LeakyReLU(0.2),
                                  nn.Linear(512, 1024),
                                  nn.LeakyReLU(0.2),
                                  nn.Linear(1024, 3*32*32),
                                  nn.Tanh()
  
                              )
  
      def forward(self, x): return self.model(x)
  
  # Define the discriminator model
  
  class Discriminator(nn.Module):
      def __init__(self):
          super().__init__()
          self.model = nn.Sequential( 
                                  nn.Linear(3*32*32, 1024),
                                  nn.LeakyReLU(0.2),
                                  nn.Dropout(0.3),
                                  nn.Linear(1024, 512),
                                  nn.LeakyReLU(0.2),
                                  nn.Dropout(0.3),
                                  nn.Linear(512, 256),
                                  nn.LeakyReLU(0.2),
                                  nn.Dropout(0.3),
                                  nn.Linear(256, 1),
                                  nn.Sigmoid()
  
                              )
      def forward(self, x): return self.model(x)
  
  # define generator training - input is fake data
  def generator_train_step(fake_data):
      # reset the gradient so the parameters will update correctly
      g_optimizer.zero_grad()
  
      # predict the output of the discriminator on fake data 
      prediction_fake = discriminator(fake_data)
  
      # torch.ones, because we want 1s outputted by the discriminator when training the generator
      error = loss(prediction_fake, torch.ones(len(real_data), 1).to(device))
      error.backward()
  
      g_optimizer.step()
      return error
  
  # define discriminator training:
  
  # discriminator as an input takes real data and fake data
  def discriminator_train_step(real_data, fake_data):
      # reset the gradient so the parameters will update correctly
      d_optimizer.zero_grad()
    
      #print(real_data.shape)
      #print(fake_data.shape)
    
      prediction_real = discriminator(real_data)
  
      # calculate loss on real data (expected 1, so torch.ones)
      real_loss = loss(prediction_real, torch.ones(len(real_data),1).to(device))
      real_loss.backward(retain_graph=True)
  
      prediction_fake = discriminator(fake_data)
    
      # calculate loss on fake data (expected 0, so torch.zeros)
      fake_loss = loss(prediction_fake, torch.zeros(len(fake_data), 1).to(device))
      fake_loss.backward(retain_graph=True)
  
      d_optimizer.step()
      return real_loss + fake_loss
  
  lr = 1e-3
  
  discriminator = Discriminator().to(device)
  generator = Generator().to(device)
  d_optimizer = optim.Adam(discriminator.parameters(), lr=lr,  betas=(0.5, 0.999))
  g_optimizer = optim.Adam(generator.parameters(), lr=lr,  betas=(0.4, 0.9))
  loss = nn.BCELoss()
  num_epochs = 50
  log = Report(num_epochs)
  
  for epoch in range(num_epochs):
      N = len(train_loader)
      for i, (images, _) in enumerate(train_loader):
          real_data = images.view(len(images), -1).to(device)
          fake_data = generator(noise(len(real_data))).to(device)
          fake_data = fake_data.detach()
          d_loss = discriminator_train_step(real_data, fake_data)
          fake_data = generator(noise(len(real_data))).to(device)
          
          d_loss = discriminator_train_step(real_data, fake_data)
          
          g_loss = generator_train_step(fake_data)
          
          log.record(epoch+(1+i)/N, d_loss=d_loss.item(), g_loss=g_loss.item(), end='\r')
      log.report_avgs(epoch+1)
      z = torch.randn(10, 100).to(device)
      # 10 pictures of dimensions of 3x32x32:
      sample_images = generator(z).data.cpu().view(10, 3, 32, 32)
      grid = make_grid(sample_images, nrow=4, normalize=True)
      show(grid.cpu().detach().permute(1,2,0), sz=10)
  log.plot_epochs(['d_loss', 'g_loss'])
  
  ```
  
  In case you would like to see the results of training this GAN, I enclose an example generated images after epoch 2:
  
  ```
  EPOCH: 2.000    d_loss: 4.936   g_loss: 1.919   (130.46s - 3131.14s remaining)))
  
  ```
  
  [![generated images after epoch 2](https://i.sstatic.net/qMfmh.png)](https://i.sstatic.net/qMfmh.png)
  
  and after epoch 50:
  [![generated images after epoch 50](https://i.sstatic.net/BETe0.png)](https://i.sstatic.net/BETe0.png)
  
  ```
  EPOCH: 50.000   d_loss: 1.253   g_loss: 1.038   (3487.47s - 0.00s remaining))
  
  ```
  
  And here is the plot of discriminator loss and generator loss:
  [![enter image description here](https://i.sstatic.net/MDJ2Q.png)](https://i.sstatic.net/MDJ2Q.png)
  
  **Tags:** neural-networks, deep-learning, generative-adversarial-networks, pytorch
  
  > **Comments:**
  > **username_1**: Did you normalize your images to [-1, 1]?  
  > **username_0**: @username_1 oh, I forgot to paste that part of code. But yes, I used transform of transforms.Compose([ transforms.ToTensor(), transforms.Normalize(mean=(0.5,), std=(0.5,)), transforms.Lambda(lambda x: x.view(-1))]) when loading data.  
  > **username_1**: It is difficult to say what is wrong with your implementation without debugging it, but GANs are known to be very sensitive to each hyperparameter, which makes the training highly unstable. For example, you don't have any normalization layers. Take a look at this [paper](https://arxiv.org/abs/1606.03498). It contains some tricks that can improve the training. But I would suggest switching to the [Wasserstein GAN](https://arxiv.org/pdf/1704.00028.pdf) or adding $R_1$ [regularization](https://paperswithcode.com/method/r1-regularization)  
  
  ---
  
  ## 0 answers
  
  