In Pytorch the optim.SGD update rule has the form:
Screenshot from 2020-05-13 16-41-461108×47 11.4 KB
Is there any reason why it is common practice to couple the learning rate with momentum (beta) and weight decay (mu)? As it is, I can’t increase momentum (smoothing of gradients) without increasing the magnitude of the velocity (v), i.e. effective step size.
Is there a good reason why Pytorch hasn’t opted for a more ‘disentangled’ update rule like this:
Screenshot from 2020-05-13 16-56-03977×34 9.05 KB
I seem to be getting a similar perf on CIFAR-10 with it, although it requires different values of mu and beta.