Solved by ptrblck in post #2 
              
              
                Yes, you can freeze parameters by setting their .requires_grad attribute to False.
The forward hook would in the simple use case just return the intermediate activation, so Autograd will not record any operation and you can just pass the tensor to any layers.