The input shape is torch.Size([128, 1, 64, 64]) the output for netD(input) is torch.Size([128, 1, 1, 1]). I just tried to use data parallelism on the WGAN code I shadowed from with my own custom train loader and 2 GPU works. I guess I just need to look at all the other changes I made which aren’t that different except the layer’s activations functions. I might just redownload Pytorch because this is currently the pre-official windows version.
Edited: NVM it also doesn’t work from the code I shadowed from I might as well just reinstall official windows version of pytorch…
Edited2: I got it working but not 100% sure… is it normal to see GPU2 usage at 0% but both GPU1/2’s clock and memeory  are utilized? I had to explicitly tell pytorch to put both netD and netG on the same GPU, at first I thought this is the same as using 1 GPU but when everything was running and double checking with evga’s PrecisionX I noticed both cards are running. Does pytorch merge 2 cards into one or something so the other card appears to be idle?
netG = nn.DataParallel(netG, device_ids=[0])
netD = nn.DataParallel(netD, device_ids=[0])