Solved by justusschock in post #2 
              
              
                How did you create your optimizer and your loss function? 
I think the problem might be, that your optimizer is created with parameters already on GPU which causes an error since its internals are (and are supposed to be) on CPU. To fix that, you could try to create the optimizer before pushing the …