Nevermind - just found out that labels are shifted inside the model and the loss for last one gets ignored.
https://huggingface.co/transformers/model_doc/gpt2.html 15

labels ( torch.LongTensor of shape (batch_size, sequence_length) , optional) – Labels for language modeling. Note that the labels are shifted inside the model, i.e. you can set labels = input_ids Indices are selected in [-100, 0, ..., config.vocab_size] All labels set to -100 are ignored (masked), the loss is only computed for labels in [0, ..., config.vocab_size]