Possibly Incorrect Perplexity Calculation in Conceptual Guide
In the docs, Conceptual Guides->Perplexity, the code underneath the section titled "Example: Calculating perplexity with GPT-2 in 🤗 Transformers" might be wrong.  This is based on my understanding.  The specific line of example code:

https://github.com/huggingface/transformers/blob/e8cc02555ee7dce7213e624ab088d8d4d1952064/docs/source/en/perplexity.mdx?plain=1#L122

I believe it should be:

`neg_log_likelihood = outputs.loss * (trg_len - 1)`

This is because `outputs = model(input_ids, labels=target_ids)` calculates `trg_len - 1` losses (and then averages them), not `trg_len`.

You can see why in the model code:

https://github.com/huggingface/transformers/blob/68287689f2f0d8b7063c400230b3766987abf18d/src/transformers/models/gpt2/modeling_gpt2.py#L1100C5-L1106

Basically, the HF API states "Note that the labels are shifted inside the model".  Because of this design choice, the model can only ever calculate `n - 1` losses, since it has to shift the labels itself.  `shift_logits = lm_logits[..., :-1, :].contiguous()` throws away the last logit, which it can't use to calculate a loss because it isn't given a label for the last position.  `shift_labels = labels[..., 1:].contiguous()` throws away the useless first label.

It's an odd decision in the API and should perhaps be a separate feature request to fix.  Regardless, this bug report is about the Perplexity calculation guide.  Since the model is only calculating `trg_len - 1` losses, it should only multiply the loss by `trg_len - 1`, not `trg_len`.

I believe the last line would also be wrong for similar reasons: `ppl = torch.exp(torch.stack(nlls).sum() / end_loc)`

An alternative is to change the guide to calculate the loss itself. This would allow the full `trg_len` labels to be used correctly.
That's correct, would you like to open a PR with your fix?<|||||>I can, yes. Which would be best: a fix for those two lines, or re-writing to calculate loss outside the model?  The latter better matches how the guide explains things but is a more extensive change and requires updating verbiage in the guide.

Additionally, is it preferred to keep the existing style of multiplying the loss inside the loop and dividing outside?  Seems like a simple `mean` outside the loop is sufficient, shouldn't result in any reduction in numerical accuracy, and is more efficient.<|||||>I think the simple fix is enough. We can also average the losses outside of the loop indeed.