Solved by albanD in post #2 
              
              
                Hi, 
Yes it is a one time thing. And the weights will stay on the GPU until you delete the model, or move it back to the cpu(). 
This comment might be related to loading of a saved model maybe? With the saved model being on CPU.