I am running the DCGAN example code at https://github.com/pytorch/examples/blob/master/dcgan/main.py 2, using the MNIST dataset. When I run with a single GPU, it behaves as expected, but when I use the parallel.data_parallel option by setting ngpu to, e.g. 2, the discriminator produces negative values, despite the last layer being a Sigmoid. I have tried moving the sigmoid after the parallel processing, and training ended after a few steps when NaNs crept in. Comparing the output with the same input using data_parallel or not shows that the outputs are not the same (or even similar) before the sigmoid layer.
I assume this is a trivial mistake I’m making, this code must’ve been used successfully by many people by now!
Set-up:

pytorch 1.0.1
cudatoolkit 9.0
cudnn 7.3.1
2x GTX 1080 Ti GPUs