Knowledge Distillation for sigmoid function I am studying the usage of knowledge distilling and all the contents found on youtube and papers suggests the use of softmax layer as the last layer because of the temperature value and the probability distribution. I'd like to know, if it means it works only with a softmax layer or can i use a sigmoid layer for binary classification on the teacher side and a softmax on the student side? I haven't found so far any example of code/description of context using a sigmoid layer on the teacher's side

As long as you calculate the terms in the cross entropy, i.e. $y_i$ for class $i$ correctly, you can use sigmoid. In the sigmoid one, you'll have one output instead of two.
