Hi all,
do I see it correctly in the source code that momentum in SGD is applied as
v = momentum * v + (1-dampening) * gradientW
W = W - lr * v

instead of the original?
v = momentum * v + (1-dampening) * lr * gradientW
W = W - v

Wouldn’t that result in applying the learning rate scaling twice to the past gradient? Is this intended or a bug?