Hi Shuvayan!



 Shuvayan:

Now in the paper, it is mentioned to take a cross-entropy loss between the outputs from the big model which will act as soft targets for the small model. However neither the size nor the data type ( FloatTensor ) is suitable for cross-entropy loss in the pytorch library . I think there’s api available for the same in tensorflow.


Yes, pytorch’s cross_entropy_loss() is a special case of cross-entropy
that requires integer categorical labels (“hard targets”) for its targets.
(It also takes logits, rather than probabilities, for its predictions.)
It does sound like you want a general cross-entropy loss that takes
probabilities (“soft tagets”) for its targets.  This general version is not
built in to pytorch.
But you can implement the general version using pytorch tensor
operations.  See this earlier thread:

  
    
    
    Soft Cross Entropy Loss (TF has it does Pytorch have it) 
  
  
    softmax_cross_entropy_with_logits 
TF supports not needing to have hard labels for cross entropy loss: 
logits = [[4.0, 2.0, 1.0], [0.0, 5.0, 1.0]] labels = [[1.0, 0.0, 0.0], [0.0, 0.8, 0.2]] tf.nn.softmax_cross_entropy_with_logits(labels=labels, logits=logits) 
Can we do the same thing in Pytorch?
  


Note that the softXEnt() implemented in this post also takes logits
for its predictions.  If your use case requires you to pass in probabilities
for your predictions (less numerically stable), you will have to modify
softXEnt() accordingly.
Good luck.
K. Frank