@Hung_Nguyen Right.
Remove the gradient penalty and the loss should still endlessly increase.
The official(?) Wasserstein GAN code doesn’t suffer from this weird behavior with parallelization, so that could be a starting point. It uses nn.parallel.data_parallel - the functional version of nn.DataParallel, but I don’t know if there’s an interesting difference there.
Converting any WGAN-GP repo into a regular WGAN mitigates the behavior somewhat, but weight clipping has its drawbacks.