Solved by ptrblck in post #2 
              
              
                Both should work, since the model is pushed to the default device and the optimization will also take place on this device. 
This blogpost by @Thomas_Wolf explains it in a nice way.