Greetings,
I am confused by PyTorchs usage of the momentum SGD method.
https://pytorch.org/docs/stable/generated/torch.optim.SGD.html#torch.optim.SGD 1
The formulas state that
v_{t+1} = \mu * v_{t} + g_{t+1} 
p_{t+1} = p_{t} - lr  * v_{t+1}

The documentation remarks that this differs from the original formula used by Sutskever et al. (download ) in terms of the application of the learning rate. However, it also differs by the fact that PyTorch subtracts the velocity from the parameter, instead of adding it.
Shouldn’t the formula be the following?
v_{t+1} = \mu * v_{t} - g_{t+1} 
p_{t+1} = p_{t} + lr * v_{t+1}

Note that in all cases, \mu > 0 is assumed.
Regards