Need correction in GPT-2 perplexity equation based on computation
# ❓ Questions & Help

I would like to formulate GPT-2 perplexity [calculation  ](https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_language_modeling.py#L266)in terms of equation as under:

![image](https://user-images.githubusercontent.com/64096105/81352017-2d8b7b80-90ce-11ea-8cdd-b44fdccf0ea3.png)

![image](https://user-images.githubusercontent.com/64096105/81352211-ab4f8700-90ce-11ea-990d-524c9756addf.png)

I am confused which equation can best describe the process. 

Following are the representation:
1. N is orignal word length
2. M is the BPE word length after tokenization
3. w represents `n_position` size, which is 1024 for GPT-2 pretrained model.
4. i=0 till M

I try to make the first equation, as GPT-2 cannot calculate perplexity in one go, if M is greater than w. It needs to calculate mean of cross-entropy loss.

Can you please share your opinion in formulating an equation?

Off-topic, but it's high time that Github supports KaTeX =)<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
