Hi Thomas,
Thanks for the detailed answer! I read through WGAN-GP and implemented the gradient penalty instead of using parameter clamping, but I’m now seeing exploding gradients. I’m using the paper’s parameters (notably lambda = 10), yet my D_real and D_fake losses are staying fairly low, on the order of -1e-1 or -1e-2, while the gradient penalty is on the order of 10. It’s actually quite strange: all of a sudden, the loss will plummet from -10 or so to -2000 and then keep jumping. Maybe it’s the use of the Adam optimizer (WGAN warns that momentum based optimizers can cause this, but their image depict much lower magnitudes). In any case, it seems I’ve traded one problem for another, while not really affecting the loss. Are you familiar with this effect with WGAN-GP?
It is worth noting that I’ve just added LayerNorm to my discriminator per the WGAN-GP paper’s suggestion (shout out to the devs who unknowingly had spot-on timing merging that PR into Master today). Perhaps that will resolve the exploding gradients (and losses going to -30k)?