Zero-shot PPLs vary wildly across gpt2 model sizes
# ❓ Questions & Help

## Details

We've been experimenting with multiple datasets for fine-tuning both OpenAI GPT and GPT-2.

What we've noticed is that the zero-shot PPL of GPT-2 small and medium are in the order of `3e+40`. However, the zero-shot PPL of GPT-2 large and XL on the same datasets are much more reasonable, like ~30. The zero-shot PPL of OpenAI GPT on the same datasets are also reasonable, on the order of 200-300.

We're trying to understand why the pre-trained GPT-2 small and medium models display such vastly different zero-shot PPLs in comparison to large, XL and OpenAI GPT.

Has anyone else also noticed the same in their experiments? What might the cause for this be?


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
