No, PyTorch doesn’t scale the gradients (unless you are using mixed-precision training with the GradScaler, but in any case the gradients would be unscaled before the update, so you can skip this side note) and you could take a look at the SGD implementation here 1 and here 1 to check this implementation and compare it against yours.