hi, thanks for the suggestion but the learning rate scheduler returns an error if last_epoch != -1 and optimizer’s state doesn’t have a ‘initial_lr’ key
The issue I’m facing may come from pytorch _LRScheduler class __init__ method, which ends with a self.step() call that may change the learning rate value…
I’m thinking about this because as soon as I instantiate the LR scheduler, the optimizer’s learning rate is modified:
# let's say we have stopped training after epoch 45 out of 100 
# and we want to resume training
epoch_restart = 45

new_optimizer = torch.optim.Adam(model.parameters(), lr=1)
# load optimizer state saved when training stopped 
new_optimizer.load_state_dict(optimizer_state)

# retrieve optimizer's lr value before instantiating LR scheduler
lr_first = new_optimizer.state_dict()['param_groups'][0]['lr']

new_scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(
    new_optimizer,
    T_max=100,
    eta_min=0,
    last_epoch=epoch_restart,
)

# retrieve optimizer's lr value after instantiating LR scheduler
lr_then = new_optimizer.state_dict()['param_groups'][0]['lr']

if you compare lr_first and lr_then, they’ll be different!
why?