Solved by tom in post #4 
              
              
                Yes, stick the parameters in two optimizers. That should not make much of a difference w.r.t. performance, you just need to zero_grad and step twice.