More details about DistilBERT experiment setting.
NIPS workshop paper (http://arxiv.org/abs/1910.01108) does not provide loss weights and other distillation hyper-paramerters (temperature, learning rate, epoch, step... ). and also (https://medium.com/huggingface/distilbert-8cf3380435b5). So when I try to experiment with this scripts(/transformers/examples/distillation/), it's hard to set hyper parameters. 

Actually, I tried to train my model with distilation/ scrpts (loss weights are set equally 0.33), and got undesirable result. :sob: I think it would be helpful for others to explicitly write experimental settings of DistilBERT in the readme file. Have a good day.
Did you take a look at the [distillation README](https://github.com/huggingface/transformers/tree/master/examples/distillation)? It shows the command that was used to train the distilled model.<|||||>Hey @silencio94, if you need help with distillation, drop me an email at clement [at] huggingface [dot] co.<|||||>Thanks for replying. I'm planing to do some experiments on my DistilBERT after distillation. after Then I'll send an email or leave another issue. thanks!