Solved by ptrblck in post #4 
              
              
                In that case amp can skip the optimizer.step(), if invalid gradients are found and you should thus also skip the scheduler.step(). 
As a workaround, you could check the scale value via scaler.get_scale() and skip the scheduler.step(), if it was decreased. We’ll provide a better API for it so that yo…