Solved by Dwight_Foster in post #2 
              
              
                Yes when you define the optimizer you can do this 
optimizer = optim.Adam(model.roi_heads.box_predictor.cls_score.parameters())

and it will only train the cls_score layer.