Default Parameters for training DistillBERT and DistillGPT2
https://github.com/huggingface/transformers/blob/cf3c9198aad5e2ea02e778aa9b04d27c216d1a35/examples/research_projects/distillation/train.py#L188

Hi @VictorSanh , 

I was going through your distillation code. Can you share what are the most suitable hyperparameters for training Distillation models, mainly DistillGPT2? Are the default parameters best to use? I am confused because the default batch size of 5 and 50 gradient accumulation step (i.e., 5X8X50 = 2000 examples) do not align with number reported in the [paper](https://arxiv.org/abs/1910.01108) (4K examples). 
 
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.