Solved by ptrblck in post #2 
              
              
                Yes, to('cuda') and .cuda() are equivalent calls. 
Make sure to push the model to the appropriate device before creating the optimizer.