Solved by ptrblck in post #2 
              
              
                Try to push the losses to the same device before accumulating them. 
Also, which PyTorch version are you using, as I cannot reproduce this issue locally (and also thought it would be fixed by now).