Solved by ptrblck in post #2 
              
              
                Your approach looks reasonable as you are not duplicating the optimizer but rather use two different optimizers and schedulers for different parts of the model. 
I think this use case looks clean and is quite easy to understand so I would stick to it and not hack around param_groups of the optimizer…