Momentum update and in-place operators in optimizers
In SGD optimizer code, <SCODE>                if momentum != 0:
                    param_state = self.state[p]
                    if 'momentum_buffer' not in param_state:
                        param_state['momentum_buffer'] = d_p.clone()
                    else:
                        buf = param_state['momentum_buffer']
                        d_p = buf.mul_(momentum).add_(1 - dampening, d_p)

                p.data.add_(-group['lr'], d_p)
<ECODE> It seems that the update in pytorch is something like this: <SCODE>v = momentum * v + (1-damping) * dp
p = p - lr * v
<ECODE> I felt just weird that learning rate is multiplied by both terms. Instead, I expect that <SCODE>v = momentum * v + (1-damping) * lr * dp
p = p - v
<ECODE> I just want to know it is a bug or intended.
Ja-Keoung_Koo
It’s intended. I think some frameworks use one definition, while others use the second one.
apaszke