Solved by MariosOreo in post #2 
              
              
                Hi, 
You could replace old_embedding.parameters() in optimizer.param_groups[0]['params'] by new_embedding.parameters() to fit your requirement.