Solved by ptrblck in post #3 
              
              
                I think your code won’t work, since the copied optimizer2 will lose its reference to the new copy of model and thus won’t update it anymore. 
You would thus have to recreate a new optimizer with model2's parameter references and load its state_dict as shown here: 
def train(model, optimizer, loss, d…