I’m investigating the use of a Wasserstein GAN with gradient penalty in PyTorch. I’m heavily borrowing from Caogang’s implementation 3, but am using the discriminator and generator losses used in this implementation 5 because I get  Invalid gradient at index 0 - expected shape[] but got [1]  if I try to call  .backward() with the  one  and  mone  args used in the Caogang implementation.
I’m training on a dataset of 400k 64x64 images, and have gotten a normal WGAN (with weight clipping to work) [i.e. it produces passable images after 25 epochs], despite the fact that the D and G losses both hover around 3. I calculate them using  torch.mean(D_real)  etc. for all epochs. However, in the WGAN-GP version, the generator loss increases dramatically (starts at ~24, then climbs rapidly to 6000 (!) in only the 6th epoch), while the discriminator loss starts at -7, decreases to -5000, then by the 6th epoch is up to +50 (!). WGAN-GP and LSGAN versions of my GAN both completely fail to produce passable images even after 25 epochs. I use nn.MSELoss() for the LSGAN version of my GAN.
I don’t use any tricks like one-sided label smoothing, and I train with default learning rats in both the LSGAN and WGANGP papers. I use the Adam optimizer and I train the discriminator 5 times for every generator update in my WGANs. Why does this crazy loss behavior happen, and why does the normal weight-clipping WGAN still ‘work’ but WGANGP and LSGAN completely fail?
This happens when using LSGAN or WGANGP irrespective of the structure, whether both G and D are normal DCGANs or when using this modified DCGAN, the Creative Adversarial Network, which requires that D be able to classify images and G generate ambiguous images. It does this through an additional K-way classification loss, for which I’m using nn.CrossEntropyLoss, and adding to D_loss.
I get erratic loss behavior (G and D losses not steadily decreasing, but instead going up and down)  in ‘normal’ DCGAN versions of my GAN, with a nn.BCELoss and the following D and G networks:
class Can64Discriminator(nn.Module):
    def __init__(self, channels,y_dim, num_disc_filters):
            super(Can64Discriminator, self).__init__()
            self.ngpu = 1
            self.conv = nn.Sequential(
                    nn.Conv2d(channels, num_disc_filters // 2, 4, 2, 1, bias=False),
                    nn.LeakyReLU(0.2, inplace=True),
            
                    nn.Conv2d(num_disc_filters // 2, num_disc_filters, 4, 2, 1, bias=False),
                    nn.BatchNorm2d(num_disc_filters),
                    nn.LeakyReLU(0.2, inplace=True),
            
                    nn.Conv2d(num_disc_filters, num_disc_filters * 2, 4, 2, 1, bias=False),
                    nn.BatchNorm2d(num_disc_filters * 2),
                    nn.LeakyReLU(0.2, inplace=True),
            
                    nn.Conv2d(num_disc_filters * 2, num_disc_filters * 4, 4, 2, 1, bias=False),
                    nn.BatchNorm2d(num_disc_filters * 4),
                    nn.LeakyReLU(0.2, inplace=True),
        
                    nn.Conv2d(num_disc_filters * 4, num_disc_filters * 8, 4, 1, 0, bias=False),
                    nn.BatchNorm2d(num_disc_filters * 8),
                    nn.LeakyReLU(0.2, inplace=True),
        
                )
      
            self.real_fake_head = nn.Linear(num_disc_filters * 8, 1)
            
            # no bn and lrelu needed
            self.sig = nn.Sigmoid()
            self.fc = nn.Sequential() 
            self.fc.add_module("linear_layer{0}".format(num_disc_filters*16),nn.Linear(num_disc_filters*8,num_disc_filters*16))
            self.fc.add_module("linear_layer{0}".format(num_disc_filters*8),nn.Linear(num_disc_filters*16,num_disc_filters*8))
            self.fc.add_module("linear_layer{0}".format(num_disc_filters),nn.Linear(num_disc_filters*8,y_dim))
            self.fc.add_module('softmax',nn.Softmax(dim=1))
        
    def forward(self, inp):
        x = self.conv(inp)
        x = x.view(x.size(0),-1) 
        real_out = self.sig(self.real_fake_head(x))
        real_out = real_out.view(-1,1).squeeze(1)
        style = self.fc(x) 
        return real_out,style

class Can64Generator(nn.Module):
    def __init__(self, z_noise, channels, num_gen_filters):
        super(Can64Generator,self).__init__()
        self.ngpu = 1
        self.main = nn.Sequential(
        nn.ConvTranspose2d(z_noise, num_gen_filters * 16, 4, 1, 0, bias=False),
        nn.BatchNorm2d(num_gen_filters * 16),
        nn.ReLU(True),
        nn.ConvTranspose2d(num_gen_filters * 16, num_gen_filters * 4, 4, 2, 1, bias=False),
        nn.BatchNorm2d(num_gen_filters * 4),
        nn.ReLU(True),
        nn.ConvTranspose2d(num_gen_filters * 4, num_gen_filters * 2, 4, 2, 1, bias=False),
        nn.BatchNorm2d(num_gen_filters * 2),
        nn.ReLU(True),
        nn.ConvTranspose2d(num_gen_filters * 2, num_gen_filters, 4, 2, 1, bias=False),
        nn.BatchNorm2d(num_gen_filters),
        nn.ReLU(True),
        nn.ConvTranspose2d(num_gen_filters, 3, 4, 2, 1, bias=False),
        nn.Tanh()
        )
    def forward(self, inp):
        output = self.main(inp)
        return output

What could be causing this?  I’d like to make as minimal change as possible, as I want to compare loss functions alone. Any help would be greatly appreciated.
Thanks in advance!