@BramVanroy That is what I was thinking! If emojis are in the vocab, which they are, they should be predicted tokens when making predictions right? Like, my vocab size with the emojis is 53575, and when making predictions, there is one 1 predicted emoji in the vocab size of 53575 - which means something is going wrong. What do you think?