Worse F1 on squad2 with finetune+distil distilroberta-base than just finetune
Hi there, 
I am trying to finetune distilroberta on squad2. First, I simply used the _distilroberta-base_ model and finetuned it on the squad2 dataset using `run_squad.py`, which gave me **74/71 F1/EM**. It's a lot worse than the roberta-base accuracy.

Currently, I am trying to finetune+distil (from roberta-base squad2 finetuned model) using `run_squad_w_distillation.py`. My roberta-base squad2 finetuned model has around 83/80 F1/EM. However, when I try to finetune+distil _distilroberta-base_ with the finetuned roberta-base as teacher, I only get around **63/60 F1/EM**. Maybe my hyperparams are way off or I need to train longer? Here's my current config:
- learning_rate=3e-5
- total_batch_size=16
- num_train_epochs=2
- max_seq_length=384

I left all other hyperparams as default.

I also checked out some predictions and it seems the model most of the time predicts _no answer_ as the best answer. In cases where it actually predicts an answer, the accuracy is not that bad.

Would be awesome to get some feedback on this, as I am trying to do inference on CPU and a distilled model would greatly benefit me in this case.

Cheers
cc @VictorSanh <|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
