I think lr decay was not applied in setting 3 and the accumulated momenta weren’t used.



 vinaykumar2491:

optimizer = torch.optim.Adam(lr=m_lr,amsgrad=True, …) if epoch % 25 == 0 and epoch != 0: lr *= decay_rate


Also, setup1 worked well w/o lr decay, I mean  for me.
So I think it’s too early to decay lr.
This is totally based on my intuition and may include a bunch of mistakes.
Hope help you.