How do you know only one GPU is working? Is the other one completely empty?
Could you add a print statement in your forward method, showing the current device of the tensor?
You can find a small example here 329. You would have to change it to print the device instead of the shape.