I am trying to Implement Spectral Normalization Paper for GAN. I removed Spectral Normalization
from  Discriminator, just to see its effect on the performance of GAN. Running this way the loss
of Generator started going haywire. I got to know that, I need to add some regularizer in to
Discriminator, so I introduced BN layer, for each CONV layer.
I am still getting losses such as this:
disc loss 0.0 			gen loss 15.3624906539917
disc loss 0.0 			gen loss 20.93044662475586
disc loss 0.0 			gen loss 21.227508544921875
disc loss 9.47e-05 		gen loss 21.953622817993164
disc loss 9.21943e-07 		gen loss 19.166460037231445
disc loss 0.0 			gen loss 11.971349716186523
disc loss 3.78e-07 		gen loss 21.656723022460938
disc loss 2.293e-08 		gen loss 24.056575775146484
disc loss 0.0 			gen loss 23.89940643310547
disc loss 6.073e-05 		gen loss 20.947172164916992
disc loss 1.82e-08 		gen loss 27.6265258789062
Where Discriminator Loss (BCE) is reaching 0, where as Generator Loss is high.
With Spectral Normalization Loss of both Desc and Generator are in range (0,2).
The Observation definitely suggests that Discriminator has become ‘too strong’. Is
there some trick which can be employed.
Regards,
Nitin BAnsal