Might help to take a look at how optimizers update parameters using the gradient. For instance, this line / block of code in SGD. https://github.com/pytorch/pytorch/blob/master/torch/optim/sgd.py#L45 897