-100 when calculating perplexity of a model..
Hi there,

I see in : https://huggingface.co/transformers/perplexity.html
there is a code block saying:


```
max_length = model.config.n_positions
stride = 512

lls = []
for i in tqdm(range(0, encodings.input_ids.size(1), stride)):
    begin_loc = max(i + stride - max_length, 0)
    end_loc = min(i + stride, encodings.input_ids.size(1))
    trg_len = end_loc - i    # may be different from stride on last loop
    input_ids = encodings.input_ids[:,begin_loc:end_loc].to(device)
    target_ids = input_ids.clone()
    target_ids[:,:-trg_len] = -100

    with torch.no_grad():
        outputs = model(input_ids, labels=target_ids)
        log_likelihood = outputs[0] * trg_len

    lls.append(log_likelihood)

ppl = torch.exp(torch.stack(lls).sum() / end_loc)
```


I am wondering why we are setting the tokens to -100. Is it a hard-coded number?
-100 is the `ignore_index` of PyTorch's `CrossEntropyLoss`, as explained in their [docs](https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html). It means that labels that are set to -100 to not contribute to the loss.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.