I am working with GPT-2 and I was looking at the LM head and how it performs the forward pass when labels are provided: https://huggingface.co/transformers/_modules/transformers/modeling_gpt2.html#GPT2LMHeadModel 57
It looks like the logits are shifted right (last value is ignored) and the labels are shifted left (first value is ignored).
Why are the logits and labels shifted in different directions?