Solved by ptrblck in post #4 
              
              
                In that case nn.DataParallel won’t help, since it would copy the data onto all GPUs. 
You could try to push S1 and S2 onto different GPUs and calculate the loss there. 
Here is a small pseudo-code for this use case: 
S1, S2 = torch.chunk(S, 2, 0)
S1 = S1.to('cuda:0')
S2 = S2.to('cuda:1')
SA1 = torch…