Can I  print mlm and nsp loss instead of their sum while train the BertForPretraning model?
The loss just contain the sum of nsp and mlm loss, I want to know their respective specific values

Is there a way to do that?
Hey @dancingpipi! There's no way to get the individual losses out of the box, but you could use the returned logits with your labels to calculate the two losses out of the model.

The loss is computed as such:

```
        if labels is not None and next_sentence_label is not None:
            loss_fct = CrossEntropyLoss()
            masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1))
            next_sentence_loss = loss_fct(seq_relationship_score.view(-1, 2), next_sentence_label.view(-1))
            total_loss = masked_lm_loss + next_sentence_los
```

and the output contains the `prediction_logits` (named `prediction_scores` above) and `seq_relationship_logits` (named `seq_relationship_score` above)<|||||>@LysandreJik thanks for your reply! 
I refer to the compute_loss, but the problem still exists: how to print them or add the to tensorboard in distributed training?
<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.