Solved by KFrank in post #2 
              
              
                Hi Siddharth! 

On the contrary, fc2.weight (and fc2.bias) are leaf tensors. 
In the forward pass the values of fc2.weight do not depend on 
the values of x (nor on the values in fc1).  They only change (in 
the conventional use case) when you call optimizer.step(). 
Are you imagining that only the …