BramVanroy:

@sgugger Shouldn’t all tokens have at least some output probability? The output predictions are over the full size of the vocabulary after all?


I don’t understand the question: if there is one prediction that is an emoji, clearly they have output probabilities. It’s just that for the sentence used ("I am feeling good today my friend "), those probabilities are lower than punctuation symbols, which seems logical to me given the fast BERT has learned to do that over a huge corpus and a long training, a habit it won’t lose in a small fine-tuning of three epochs over 1700 samples.