the lr_scheduler ReduceLROnPlateau is there just for updating the lr which should be normally called after a pass of a epoch, while all the other codes remain same i.e. optimizer.zero_grad() to reset the gradient to zero and before performing optimizer.step().