Solved by ptrblck in post #2 
              
              
                Your code looks generally alright besides this line of code: 
out2.to(gpu1)

While nn.Modules are transferred inplace, you have have to assign tensors back: 
out2 = out2.to(gpu1)

Could you fix this line and see it it’s working?