Solved by glaringlee in post #2 
              
              
                @farmersrice 
check this issue. 
‘https://github.com/pytorch/pytorch/issues/15426’, I think our document need update. 
You can not copy memory from CPU to GPU directly. Your temp[] is not on GPU. 
I think you have to use .to(device) at this point.