How much of an improvement is DistilGPT-2 over an equivalent model trained without distilation?
Hi! 

I'm working on a distillation project right now, and I was wondering if this information is available anywhere.

I saw [this page](https://github.com/huggingface/transformers/tree/9ee66adadb2a8d6e04e8b18a1c9ea0b57c80642e/examples/research_projects/distillation) provides a comparison for `DistilGPT-2` vs `GPT-2`, but I don't see anything about the improvement of `DistilGPT-2` over an equivalent model (same parameters, etc.) trained in a traditional fashion. 

Any help would be greatly appreciated. Thanks!
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.