Thank you for your help. I’m actually doing the same as you suggest in your solution:
Initial:
best_model_wts = copy.deepcopy(model.state_dict())
best_optim_state = optimizer.state_dict()

Updating in the loop:
if epoch < 20:
    error_sigma = 2.0
    elif 19 < epoch < 40:
        error_sigma = 1.5
        if epoch == 20:
            model.load_state_dict(best_model_wts)
            optimizer = optim.SGD(model.parameters(), lr=0.001)
            optimizer.load_state_dict(best_optim_state)

And saving the best models and optimizer state:
if phase == 'val':
    val_acc_history.append(epoch_acc)
    if epoch_acc > best_acc:
        best_acc = epoch_acc
        best_epoch = epoch
        best_loss = epoch_loss
        best_model_wts = copy.deepcopy(model.state_dict())
        best_optim_state = optimizer.state_dict()

However, I’m still getting the same error during the update line for optimizer.
I tried your solution and it runs without any problem on my environment (python 3.7.5, pytorch 1.2) but mine is still giving the above-mentioned runtime error.