Solved by ptrblck in post #2 
              
              
                If you are using a momentum in the optimizer and had previous valid updates for these parameters, they’ll still be updated as seen here: 
# no momentum
x = torch.ones(1, 5, requires_grad=True)
optimizer = torch.optim.SGD([x], lr=0.1, momentum=0.)

# perform a valid update
print('no momentum, valid u…