crcrpar:

I think lr decay was not applied in setting 3 and the accumulated momenta weren’t used.


I agree. Thats why I took care of externally decaying learningrate and using the accumulated momentum in Setup-4. But, I still donot understand the results of Setup-4.