Solved by ptrblck in post #2 
              
              
                You would have to push the calculated losses to the same device before summing them. 
This should work: 
loss = loss_fn(outputs1, labels1) + loss_fn(outputs0, labels0).to('cuda:1')