Perplexity VERY high but generated text coherent
I have trained GPT2 from scratch on a minority language which I tokenized with BPE. 

When I finished training, the perplexity on the eval set was `678.80`, so I thought the model was really bad.

I however tried to sample from the model using very different sampling procedures, and finally got very coherent texts with the following parameters:

`model.generate(input_ids, do_sample=True, top_k=950,  repetition_penalty=1.2, eos_token_id=0)`

- Is the model generating good texts because my top_k is larger than the perplexity?
- Is the perplexity perhaps so high because my vocab size is 50256?

Is there perhaps a bug in how the perplexity is computed after training?  This is the final value I got from the last epoch and also stored in `eval_results.json`.

Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co) instead?

Thanks!<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.