I use softmax and then I use log to mimic cross entropy when I try to implement knowledge distillation.
soft_old_output = soft_target.clone()/T
soft_new_output = outputs.clone()/T
soft_new_output.retain_grad()
outputs_S = F.softmax(soft_new_output,dim=1)
outputs_S.retain_grad()
outputs_T = F.softmax(soft_old_output,dim=1)
loss2 = outputs_T.mul(-1*torch.log(outputs_S))
loss2 = loss2.sum(1)
loss2 = loss2.mean()
loss = loss2
loss.backward(retain_graph=True)
print(soft_new_output.grad)

At first the soft_old_output is the same as soft_new_output . So the grad of soft_new_output is all zero. It’s right.
But when I try to use some of the value to do is, like that:
soft_old_output = soft_target.[:,:2]clone()/T
soft_new_output = outputs[:,:2].clone()/T
soft_new_output.retain_grad()
outputs_S = F.softmax(soft_new_output,dim=1)
outputs_S.retain_grad()
outputs_T = F.softmax(soft_old_output,dim=1)
loss2 = outputs_T.mul(-1*torch.log(outputs_S))
loss2 = loss2.sum(1)
loss2 = loss2.mean()
loss = loss2
loss.backward(retain_graph=True)
print(soft_new_output.grad)

the grad of soft_new_output will be
tensor([[-5.8213e-08, -1.3913e-09]])

Why aren’t they zero?