So , i have been trying to implement the distilled model concept by Hinton et. al. in the paper Hinton Dark Knowledge.
Accordingly i trained a cumbersome model , and depending on the results on the cumbersome model, i have to train the smaller model to fit the data.
Now the output from the big cumbersome model is of the shape ( batch_size , outputs ) which is same as the as the output size from the small model.
Now in the paper, it is mentioned to take a cross-entropy loss between the outputs from the big model which will act as soft targets for the small model. However neither the size nor the data type ( FloatTensor ) is suitable for cross-entropy loss in the pytorch library . I think there’s api available for the same in tensorflow.
I have tried KL Divergence loss for the loss between outputs of the big model and the small one and the CrossEntropyLoss for the outputs of the small model and the actual targets ( hard labels), but it gave me very poor generalizing errors, it seems to overfit very much.
How can i implement a custom loss function for the outputs from the big model ( soft targets ) and the outputs from the big model which is of the same size .
or
output = torch.rand( 2 ,4 ) 
soft_targets = torch.rand( 2 , 4 ) 
labels = torch.rand( 2 ,1).squeeze()
loss = criterion1( output , soft_targets )  + criterion2( output , labels ) 


I think criterion2 can be CrossEntropyLoss, what can be criterion1 in this case, please don’t suggest KL Divergence .