About knowledge distillation in teacher-student networks After calculating overall loss, we backpropagate this loss through the student network. But we do not backpropagate through teacher network (I am not sure from that). And when a new batch arrives, we once again forward propagate the next batch through both two networks and so on.
My question is: do we backpropagate loss through the teacher or not? If yes, what is the logic of using a student network? Or teacher network is a network trained before and just used in inference mode while training student? Can anyone clarify this, please?


Or teacher network is a network trained before and just used in inference mode while training student?

Yes, exactly. The goal of knowledge distillation is to condense a complex 'teacher' model into a simpler 'student' one, with minimal degradation of performance. You train the student model on the teacher model's predictions, rather than the true labels. This doesn't require updating/altering the teacher model.

It is only to make lower the cost of the test (since we use a shallower model to test)?

Yes, that's right: you do this for faster or less computationally demanding inference. (There are other fringe reasons. Maybe you also want a more interpretable, shallower model, or maybe you have the fitted teacher model but either (a) no longer have the true labels, or (b) are trying to fit to a new dataset.)
