Oh. I was using Adam! Thanks for clarifying.
A not so thorough reading of, and lack of resolution in, old learning rate updation queries like this 10 might be behind such confusion 2.
Maybe the optimizers where the internal state matters should be tagged at the torch optim 7 page? Also, possibly at saving and loading models for inference 7. Thanks again for the prompt response.