@pietern even if the gradients didn’t accumulate properly, it should at worst look as if I only have one GPU. I’ve verified that everything works well with up to 4 GPUs with pytorch 0.4.1. But I can’t seem to nail down the actual problem in 1.0
I’m not even sure how to diagnose this, but I’ve been able to replicate this behavior with dataparallel on 2 other popular WGAN repos from github.