I tried all these different ways actually, the error remains. Optimizer is SGD with momentum:
optimizer_ft = optim.SGD(model_ft.parameters(), lr=params.lr, momentum=params.momentum)
optimizer_ft.zero_grad()

I also tried without copy.deepcopy as in your solution:
best_model_wts = copy.deepcopy(model.state_dict())
best_optim_pars = optimizer.state_dict()

And then, in the code when I load:
 if epoch == 20:
     model.load_state_dict(best_model_wts)
     #this is for initialization, not sure if needed?
     optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.9)
     optimizer.zero_grad()
     optimizer.load_state_dict(best_optim_pars)

For saving the best model and the current optim params:
 if phase == 'val':
     val_acc_history.append(epoch_acc)
     if epoch_acc > best_acc:
         best_acc = epoch_acc
         best_epoch = epoch
         best_loss = epoch_loss
         best_model_wts = copy.deepcopy(model.state_dict())
         best_optim_pars = optimizer.state_dict()

Still getting this error:
File “/media/alic/ssdmain/Projects/crandrnn/src/train_multi_level_model.py”, line 83, in train_model
optimizer.load_state_dict(best_optim_pars)
File “/home/alic/anaconda3/envs/crandrnn/lib/python3.7/site-packages/torch/optim/optimizer.py”, line 105, in load_state_dict
File “/home/alic/anaconda3/envs/crandrnn/lib/python3.7/site-packages/torch/tensor.py”, line 23, in deepcopy
raise RuntimeError("Only Tensors created explicitly by the user "
RuntimeError: Only Tensors created explicitly by the user (graph leaves) support the deepcopy protocol at the moment.