that probably should, but again I do not see it… having that there is no momentum, or weight decay, just pure SGD with lr, where
model.layer.weight -= lr * model.layer.weight.grad
model.layer.bias -= lr * model.layer.bias.grad

takes place more precisely?