How to download original weights of gpt2
# ❓ Questions & Help
Hi, I have been trying to reproduce losses from [this paper](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf), but I realized that my losses are lower than the ones from the paper. Also, I noticed a warning - Some weights of GPT2LMHeadModel were not initialized from the model checkpoint at gpt2-medium and are newly initialized: ... Does it mean that the original weights of GPT-2 was changed? If so, how can I download the original weights?
For instance my PPL for PTB is 35.20 while the PPL in the paper is 65.85.
Hey @nuradilK,

1) The GPT2 weight initialization is a known warning bug we should remove, but which does not affect model performance, 
see https://github.com/huggingface/transformers/issues/5800 e.g.

2) It's very hard for us to track down low performance of specific user code (which data to use, data processing, ...). Do you mind posting a question like "How to reproduce GPT2 PPL results" on https://discuss.huggingface.co/ since it is not really a bug but more a high level question. <|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
