Solved by albanD in post #2 
              
              
                Hi, 
You can go through every parameters that you don’t want to update and set them not to require gradients: 
for p in layer2.parameters():
    p.requires_grad = False

This means that no gradients will be computed for these parameters. 
As further optimization, you can give to your optimizer only …