Can you also try saving the scheduler state_dict, and then load that along with model and optimizer states?
model = nn.Conv2d(3, 8, 3)
optimizer = optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
sched = lr_scheduler.MultiStepLR(optimizer, milestones=[10, 20, 30], gamma=0.1, last_epoch=-1
)
>>> for i in range(25):
...      sched.step()

## current state of sched: and optimizer:
>>> optimizer
SGD (
Parameter Group 0
    dampening: 0
    initial_lr: 0.1
    lr: 0.0010000000000000002
    momentum: 0.9
    nesterov: False
    weight_decay: 0
)
>>> lr_sched.state_dict()
{'milestones': [10, 20, 30], 'gamma': 0.1, 'base_lrs': [0.1], 'last_epoch': 24}

## saving the checkpoint:
>>> torch.save({'model_state_dict': model.state_dict(),                                                 
                'optimizer_state_dict': optimizer.state_dict(),
                'scheduler_state_dict': sched.state_dict()}, 'checkponit.pth') 


now we can load all of them:
## loading the checkpoint
>>> checkpoint = torch.load('checkponit.pth')

>>> model_2 = nn.Conv2d(3, 8, 3)                                                                      
>>> optimizer_2 = optim.SGD(model_2.parameters(), lr=0.1, momentum=0.9)
>>> sched_2 = optim.lr_scheduler.MultiStepLR(optimizer, milestones=[10, 20, 30], gamma=0.1, last_epoch=-1)

>>> model_2..load_state_dict(checkpoint['model_state_dict'])
>>> optimizer_2.load_state_dict(checkpoint['optimizer_state_dict'])
>>> sched_2.load_state_dict(checkpoint['scheduler_state_dict'])
>>> optimizer_2
SGD (
Parameter Group 0
    dampening: 0
    initial_lr: 0.1
    lr: 0.0010000000000000002
    momentum: 0.9
    nesterov: False
    weight_decay: 0
)
>>> sched_2.state_dict()
{'milestones': [10, 20, 30], 'gamma': 0.1, 'base_lrs': [0.1], 'last_epoch': 24}

So this shows that the 'last_epoch ’ is correctly loaded. Now, we can just try one more epoch for final verification:
>>> sched_2.step()
>>> sched_2.state_dict()
{'milestones': [10, 20, 30], 'gamma': 0.1, 'base_lrs': [0.1], 'last_epoch': 25}
>>> optimizer_2
SGD (
Parameter Group 0
    dampening: 0
    initial_lr: 0.1
    lr: 0.0010000000000000002
    momentum: 0.9
    nesterov: False
    weight_decay: 0
)

So now as you can see, the lr of the optimizer is not re-initialized.