Unable to match GPT-2 reported perplexity results on CBT, Wikitext-103, and 1BW datasets and doubt on LAMBADA accuracy
I am having the following discrepancies on trying to reproduce the results of GPT-2 small (117M parameters).

On using this huggingface [script for perplexity](https://huggingface.co/transformers/perplexity.html). I am able to match reported perplexity on Wikitext-2 (reported - 29.41, reproduced 29.94). But it also gives the same perplexity on Wikitext-103. Likewise [EleutherAI's language model evaluation harness](https://github.com/EleutherAI/lm-evaluation-harness) gives perplexity that matches Wikitext-103's reported perplexity (reported 37.50, reproduced - 37.37) and gives the same result for Wikitext-2. The methods give same perplexity to Wikitext-2 as well as Wikitext-103 as the test set for the datasets are the same. The huggingface method uses the actual token count and the EleutherAI one uses word count to calculate perplexity. Is the assumption correct that the different perplexities are simply due to the different ways they are computed?

I am also unable to match Children's Book Test (CBT) dataset tasks CBT-CN and CBT-NE accuracies of 87.65 and 83.4. I am able to get up to 84.35 and 73.65 only.

Getting accuracy of 46.67 on LAMBADA dataset for predicting the last token instead of the last word. Is this way correct?

Getting perplexity of 45.88 using token count and 89.35 using word count on one billion words (1BW) dataset instead of the reported 75.20.
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>hi @samzabdiel, did you ever find what was going on? <|||||>Not completely, but I was able to closely match accuracies reported on CBT datasets using the appropriate preprocessing. The results seem to be tightly coupled with the preprocessing done.