@ptrblck is right, it’s coming from accumulating losses, a simple repro is
a=torch.tensor(2.)
b=torch.tensor(2., device="cuda")
a += b

The workaround is
a = a + b

(a would be on the gpu in this case)
I’ve opened an upstream issue to track

  
      github.com/pytorch/pytorch
  
  
    
  
	  
  

  
    
      In-place operation between cpu scalar and gpu scalar throws an error 107
    

    
      
        opened 
        
          
        
        Jul 5, 2020
      
      


      
        
          
          ngimel
        
      
    
  



  🐛 Bug
To Reproduce
a=torch.tensor(2.)
b=torch.tensor(2., device="cuda")
a += b
results in
RuntimeError: Trying to pass too many CPU scalars to CUDA kernel!
Expected behavior
No error (that's tricky...



    topic: TensorIterator
    triaged