Hi Kale, thanks, just caught with your discussion on youtube comment thread 39  and I am
glad to learn that there is a checker built in pytorch that raises Error if we try to set requires_grad=False for interim nodes.
In summary:

we do not (and can not) set requires_grad=False on intermediate layers that we want to freeze because their gradients are needed for calculating gradients of other connected nodes (i.e. the definition of a leaf node in grad network).
but intermediate nodes that we want to freeze can be excluded from the optimizer.

So,

  
    
    
    Freezing intermediate layers while training top and bottom layers autograd
  
  
    maybe, in my case, I should not be setting requires_grad=False to the L2 parameters,  instead I must exclude all L2 parameters from optimizer. That way, right amount of gradients will flow back to L1’s params, but optimizer does not update L2 parameters (which is analogous to freezing L2, yet keeping L1 trainable) 
Is this a correct approach  ?
  


is the right approach!
let me go and try this stuff.
Cheers and thanks for the help,
TG