@apaszke the concern here is definitely about losing the running averages, etc maintained by the optimizer. As you say the cost of constructing it is negligible.
In the models I’m training right now I see an increase in the loss when I construct a new optimizer to decay the learning rate.