Hello Wmog (and others) -
I had the exact same question / confusion, so I’m glad I found this
thread.  (Sorry to revive an oldish thread …)



 Wmog:

Ok perfect, that was exactly what I thought. Actually, they should be named “Stepper”. For example with SGD that will be “SGDStepper”. That seems more clear.


I agree with your analysis.  torch.optim.SGD doesn’t have any
randomness (stochasticity) in it, so it does seem misleading to
call it stochastic gradient descent.  Calling it gradient descent
(or GD) would be more accurate.  Or, as you note, perhaps calling
it “gradient descent optimization stepper” would emphasize that
it only performs one step at a time.  (Maybe folks thought that
leaving “stochastic” out of the name would confuse people who
just wanted to use the “standard” SGD optimization algorithm.)
Best.
K. Frank