oh that’s too bad. Unfortunately, I don’t have any better idea how to tackle this case.

Setting output to GPU 3 seems to only collect the loss to that GPU. Gradients seem to still be gathered in GPU 0.

I think this would be a useful thing to get to work properly such that the gradients are also gathered on the output device. I am not sure if this is a bug or a limitation, but I think this could be a useful feature/fix in PyTorch. Maybe that could be posted on the issue tracker for further discussion/investigation.