Solved by albanD in post #4 
              
              
                Moving to the gpu is recommended to be done first. 
For example, it should be done before passing the parameters to the optimizer as well.