Q / Note: BERT Masked-LM fails to predict last token in sequence if it is not punctuation
## ❓ Questions & Help

I am playing with BERT to see what the distributions of the prediction for a MASK token are. I wrote a quick script that successively masks all words in an input sequence.
This is based on the implementation in the examples (e.g. the lm finetuning script and the examples in the  documentation).

In doing so, I found out that BERT generally fails to predict the last word in the sentence if it is not punctuation. With overwhelmingly high likelihood, BERT expects a normal SVO sentence to end with a full stop. While it can predict the correct word (the correct word usually appears in the top 10 most likely tokens), the likelihood as given by softmax is very low.

So by itself this is perhaps not surprising, because the large majority of examples in pre-training will have punctuation, especially if pre-training is not just the MLM but also the sentence prediction.

But I wonder if it should be best-practice to ensure every sentence is punctuated? If the MLM part of BERT consistently predicts punctuation, then a sentence without it will not be efficiently classified compared to one with punctuation, even on downstream tasks, right?

One thing to confirm, of course, would be that this is not an issue of the pyTorch implementation and how attention masks the <SEP> token or something?

What do you think?




Attached is the script, you should just be able to run it.

[lm_test.py.txt](https://github.com/huggingface/transformers/files/3739434/lm_test.py.txt)

If you play with the script a bit, you can see that the loss for BERT with the MLM head is actually quite high, as someone suggested elsewhere, this may be due to pre-training on different tasks than just MLM<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
