Hi @Florian_1990,
As stated in the original Adam paper, Adam calculates learningrate for each parameter and updates it with each iteration. So inherently I should NOT be able to use learning rate scheduler lr_scheduler.MultiStepLR() on optim.adam() with a new learning rate.
My doubts:


In Setup-3 I’m intentionally creating a new Adam optimizer every 25 epochs but in Setup-4, I’m using available lr_scheduler.MultiStepLR() to decay learning rate every 25 epochs. And surprisingly the behavior in both these settings are same, which I’m not able to understand.


Normally PyTorch should throw a warning/error when using lr_scheduler on optim.adam() stating that external lr_decay creates inconsistent behavior or something like this.


My goal is to learn faster when the loss starts saturating [using Adam] and decaying learning rate is one of the solutions usually.
Thanks