Solved by ptrblck in post #2 
              
              
                Yes, you would need to create a new optimizer for the new model, as the old optimizer stores references to the initially passed parameters. 
Alternatively, you could add the new parameters via add_param_group, but I don’t think you would have any benefit from it.