For a word that is divided into multiple tokens by bpe or sentencepiece like model, you use the first token as your reference token that you want to predict. Since all the tokens are connected via self-attention you won’t have the problem not predicting the rest of the bpe tokens of a word. In PyTorch, you can ignore computing loss 3 (see ignore_index argument) of those tokens by providing -100 as a label to those tokens (life is so easy with pytorch  ).