The logits are not shifted, just the last value is ignored. The labels are shifted inside the model as described in the docs ("Note that the labels are shifted inside the model, i.e. you can set labels = input_ids") because we want to avoid any processing on them and just set them equal to the inputs. That way the batch creation can be as easy as possible. The downside is that you don’t compute the loss on the last character of the sentence, but we found it’s acceptable since the sequence length is 512.