Continuing the discussion from Weighted sum of two losses can not reduce to the loss of one component:
Hi,
I throw away the built-in KLDivLoss. Instead, I wrote my own KLD. I think in knowledge distillation, you can use the part in the KLD loss which is dependent on your student model only, and throw away the other part.
The original KLD loss is:
P_teacher * log(P_teacher / P_student)
I use:
P_teacher * log(1/P_student)
since this is the term related to your student model. In fact, this modified KLD is what most paper uses.
You can implement this loss in a module, just like how you define a network. I did not see your problem with the modified KLD.