I want to remove the momentum as well, that is why I want to reset it completely. Currently my code is kinda ugly coz I explicitly replace the optimizer reference and re-initialize the scheduler too with new optimizer.
It would be nice if we can reset directly from the lr_scheduler or call opt.reset().