Hey!
I just have a quick question.
I got a bit confused after having read this thread:

  
      github.com/pytorch/pytorch
  
  
    
  
	  
  

  
    
      optimizer load_state_dict() problem? 3
    

    
      
        opened 
        
          
        
        Sep 22, 2017
      
      

        
          closed 
        
          
        
        Nov 28, 2017
      
        

      
        
          
          JianyuZhan
        
      
    
  



  Hi, I encountered this bug:
 optimizer.step()
 exp_avg.mul_(beta1).add_(1 - beta1, grad)
TypeError: add_ received an invalid combination of arguments - got (float, torch.cuda.FloatTensor),...



    awaiting response (this tag is deprecated)
    needs reproduction


  
  
    
    
  
  


It’s quite old now and I saw things change quickly here!
My question is (I didn’t really find much about it on the forum or the doc): does the optimizer have to be set onto CUDA?
I know you must set the model onto CUDA before calling the optimizer (therefore its state_dict is put onto CUDA I guess), and set the inputs onto CUDA too. But what about the optimizer? It does have a state_dict but I found nothing except the linked example about it. Does it mean optimization computations are always performed on the GPU (or maybe CPU)? Are the optimizer.state.values the same as its state_dict?
I’m quite curious so if someone knows what determines the device used for computations… I know that model.cuda() sets the optimizable parameters onto CUDA and it just works, but what if I tried to put half the layers’ parameters on CUDA and half on CPU? Where would the computations then happen?
Bonus question: according to this other thread 4 it looks like constructing the optimizer before model.cuda() works just fine except for optimizers that use a “buffer” (dunno what this buffer is) which makes me think the optimizer “lives his own life” no matter the place the model is stored into…
Thanks in advance,
Regards,
Florent