Solved by albanD in post #2 
              
              
                Hi, 
By default pytorch will differentiate your whole program. 
If there are parts where the gradients should not flow back, you should make it explicit by doing t_without_grad_tracking = t.detach(). 
Here in particular, if you teacher uses the output of your student’s net for training, but the grad…