Reproducibility problem with DistilBERT paper 
## ❓ Questions & Help

We are currently working a follow-up to your work “DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter”. We’ve noticed some skeptical data in Table 1 of your paper. On MRPC, the reported averaged F1 and acc result is 90.2, which is even ~2% higher than BERT-base (teacher). We carefully reproduce your experiment with your code and pretrained checkpoint provided in huggingface/transformers on Github. Our reproduced result is 89.6/85.5, which means the averaged F1 and acc should be 87.55, which is very different from your reported result. With all due respect, we personally think you may have mistakenly report the F1 score instead of averaged F1 & acc. Another evidence is your previous blog (https://user-images.githubusercontent.com/16107619/64210993-c0ef1b80-ce72-11e9-806b-171313e8ae9e.png) and DistilRoBERTa, which has a much lower MRPC score of 86.6 (https://github.com/huggingface/transformers/tree/master/examples/distillation). We list your reported results and our reproduced results and reproduced results on GLUE dev set:

DistillBERT on GLUE Dev Set | CoLA | MNLI-m | MNLI-mm | MRPC | QNLI | QQP | RTE | SST-2 | STS-B
-- | -- | -- | -- | -- | -- | -- | -- | -- | --
DistilBERT Blog | 42.5 | 81.6 | 81.1 | 85.35(88.3/82.4) | 85.5 | 89.15(87.7/90.6) | 60.0 | 92.7 | 84.75(84.5/85.0)
DistilBERT paper | 49.1 | 81.8 |   | 90.2 | 90.2 | 89.2 | 62.9 | 92.7 | 90.7
Our reproduced | 43.0 | - | - | 87.55(89.6/85.5) | 85.8 | - | - | - | 80.53(80.6/80.5)

According to our experiment, the result is actually very close to the previous results you reported on your blog. We are not able to reproduce results reported in your paper though we have tried some hyperparameter tuning. We will really appreciate it if you can confirm the result in your paper or send us the hyperparameters to reproduce the results.


Does anyone have the same problem here?