Currently also experiencing this issue. The gradient penalty eventually goes to the millions before blowing up completely. @neale, I have tried to reproduce this issue with various popular WGAN-GP repos as well, and they also suffer from this.
Pytorch 1.0.1 on CUDA 10.1.