How important is it to use the same optimizer when continuing training?


If I train a model with an Adam Optimizer wrapped in a OneCycle LR Schedule, for let’s say ~10 epochs.


Then decide to fine-tune the model and I make a new optimizer with different settings (e.g. learning rate), and train for let’s say another ~10 epochs.


Example:
# initial model, opt, and scheduler.
my_model = torchvision.models.resnet18()
my_optim = optim.Adam(lr=1e-3)
my_lr_scheduler = optim.lr_scheduler.OneCycleLR(optimizer=my_optim, max_lr=3e-2)

# Fit/train for 10 epochs
fit()

# Fine-tune model with a smaller lr.
my_optim = optim.Adam(lr=1e-5)
my_lr_scheduler = optim.lr_scheduler.OneCycleLR(optimizer=my_optim, max_lr=3e-4)

# Fit/train for 10 more epochs
fit()

Will this perform worse compared to changing the original optimizer’s hyperparameters manually via my_optim.param_groups[0]['lr'].
I noticed that an optimizer’s state dict, my_optim.state_dict(), holds two keys: state and param_groups. Making a new optimizer in the example above will make the param_groups hold the right values, but the state is lost. The below example maintains state
Example:
# initial model, opt, and scheduler.
my_model = torchvision.models.resnet18()
my_optim = optim.Adam(lr=1e-3)
my_lr_scheduler = optim.lr_scheduler.OneCycleLR(optimizer=my_optim, max_lr=3e-2)

# Fit/train for 10 epochs
fit()

# Fine-tune model with a smaller lr.
my_optim.param_groups[0]['lr'] = 1e-5
my_lr_scheduler = optim.lr_scheduler.OneCycleLR(optimizer=my_optim, max_lr=3e-4)

# Fit/train for 10 more epochs
fit()

Is there any resources out there showing the importance of whether or not to reuse Optimizers and their states?