This is intentional, see note https://github.com/pytorch/pytorch/blob/master/torch/optim/sgd.py#L28-L48 10. Formulations are equivalent for a constant learning rate, and the one used in pytorch makes step directly proportional to learning rate when learning rate changes.