GPT2 (117M) results mismatch on PTB, enwik8 and text8 metrics
Model used: GPT2 117M parameter model
Platform: Google colab with GPU

I am trying to reproduce the results of GPT2 (117M) on various datasets but getting different values of various metrics. I have used this script from [Huggingace platform](https://huggingface.co/transformers/perplexity.html).

1. On PTB, I have used one from the source: https://github.com/wojzaremba/lstm/tree/master/data
Hopefully this is the original dataset used in GPT2 paper. **Reported value in GPT2 paper: 65.85 PPL**. I am getting the following results:
52.53 PPL (on test set using token count)
79.65 PPL (on test set using word count)
60.08 PPL (on validation set using token count)
**Can anyone clarify whether token or word count is used while calculating PPL on PTB dataset?**

2. On enwik8 dataset, **reported metric BPB value is 1.16** while I am getting a value of 2.23 using stride count of 1024. Changing stride to 512 is not having much of an affect.

3. On text8 dataset, **reported metric BPC value is 1.17** while I am getting a value of 2.29 using stride count of 1024. Again, changing stride to 512 is not improving the result.

**Is there some pre-processing that needs to be done which is not mentioned in the paper for all of the above?**
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co) instead?

Thanks!<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.