Solved by ptrblck in post #2 
              
              
                You can pass subsets of parameters to an optimizer while creating it, which makes sure that the optimizer only updates these parameters.