Error in Calculating Sentence Perplexity for GPT-2 model
Hi,

I am using a following code to calculate the perplexity of sentences on my GPT-2 pretrained model:
       
```
       tokenizer = GPT2Tokenizer.from_pretrained('gpt-model')
       config = GPT2Config.from_pretrained('gpt-model')
       model = GPT2LMHeadModel.from_pretrained('gpt-model', config=config)
       model.eval()
       
  def calculatePerplexity(sentence,model,tokenizer):
        input_ids = torch.tensor(tokenizer.encode(sentence)).unsqueeze(0) 
        input_ids = input_ids.to('cpu')
        with torch.no_grad():
            outputs = model(input_ids, labels=input_ids)
        loss, logits = outputs[:2]
        return math.exp(loss)
```
For some of the sentences from my testing corpus, I am getting following error:

**Token indices sequence length is longer than the specified maximum sequence length for this model (1140 > 1024). Running this sequence through the model will result in indexing errors**

How can I resolve this error? Kindly advise.
The longest input length a pretrained GPT2 model can treat depends on its `n_position` value. You can look it up here e.g. https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-config.json . 
If you use a pretrained-model you sadly can only treat sequences <= 1024.

For you own model you can increase `n_position` and retrain the longer position encoding matrix this way. 

If you are just interested in the perplexity you could also simply cut the input_ids into smaller input_ids and average the loss over them. It will not exactly be the same, but a good approximation.<|||||>Is it being calculated in the same way for the evaluation of training on validation set?

Secondly, if we calculate perplexity of all the individual sentences from corpus "xyz" and take average perplexity of these sentences? will it be the same by calculating the perplexity of the whole corpus by using parameter "eval_data_file" in language model script?<|||||>1) Yes 

2) No -> since you don't take into account the probability `p(first_token_sentence_2 | last_token_sentence_1)`, but it will be a very good approximation.

Hope this answers your question