BERT LOSS FUNCTION
My question is that can I use KLDivLoss instead of CrossEntropyLoss when I fine-tune BERT for classification? the reason for that is that I want to pass the weight of each class(e.g for binary classification, instead of 1 or 0 I will pass the probability distribution )



Thank you in advance
Sure you can do that. Create a class which inherits from [BertForSequenceClassification](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L1122) and overwrite the [forward](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L1134) method.<|||||>Instead of overwriting the forward method you can retrieve the hidden states and compute the loss as you would do with any PyTorch model. 

The loss is only computed by the model when you hand the `labels` to the model, which is not a required argument.<|||||>> Instead of overwriting the forward method you can retrieve the hidden states and compute the loss as you would do with any PyTorch model.
> 
> The loss is only computed by the model when you hand the `labels` to the model, which is not a required argument.

Could you please elaborate on the same please? @LysandreJik 
<|||||>What do you want me to elaborate on?<|||||>> Instead of overwriting the forward method you can retrieve the hidden states and compute the loss as you would do with any PyTorch model.
This @LysandreJik <|||||>I got this error when I called 
loss.backward()
 loss is "torch.float64" type


![error-KLDiv](https://user-images.githubusercontent.com/55197626/81520607-c7436a80-9312-11ea-91ae-85fb7f032d96.PNG)
<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>> My question is that can I use KLDivLoss instead of CrossEntropyLoss when I fine-tune BERT for classification? the reason for that is that I want to pass the weight of each class(e.g for binary classification, instead of 1 or 0 I will pass the probability distribution )
> 
> Thank you in advance

Hi did you manage to do this? I also need to pass class probability distribution instead of the labels and am not sure how to do this.<|||||>BertForSequenceClassification.forward() returns the logits also. You can use these in any pytorch loss function (eg: KLDivLoss, not sure if you'll need to softmax them first) and then run backward on the resulting loss. It's a bit redundant (since BertForSequenceClassification's loss is still calculated), but works.<|||||>> Sure you can do that. Create a class which inherits from [BertForSequenceClassification](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L1122) and overwrite the [forward](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L1134) method.

That link to the forward function is stale, as of Feb 24 it's [here](https://github.com/huggingface/transformers/blob/7e662e6a3be0ece455b4c4ae2c3348beab11bad5/src/transformers/models/bert/modeling_bert.py#L1475).