Solved by ptrblck in post #2 
              
              
                Have you tried to set the eps argument of your ReduceLROnPlateau scheduler? 
Adam’s eps is different to the one used in the scheduler. 
From the docs: 

eps ( float ) – Minimal decay applied to lr. If the difference between new and old lr is smaller than eps, the update is ignored. Default: 1e-8. 

I…