Solved by SimonW in post #2 
              
              
                The former is preferred when you want to train this parameter. Because if you do var_x = Variable(x, requires_grad=True).cuda(), the gradients are actually copied from gpu to cpu, and accumulated to the Variable(x, requires_grad=True) term, which you don’t save. So it is lost. 
In other cases it sho…