Solved by ptrblck in post #4 
              
              
                Are you using the GradientLoss inside the nn.DataParallel module? 
If so, remove the cuda() operations from the GradientLoss.__init__ method, register the tensors as buffers via torch.register_buffer, and nn.DataParallel should be able to clone this module to all passed GPUs.