End-Task Distillation with DistilBERT
## ❓ Questions & Help

The DistilBERT paper notes the IMDB and SQuAD results were obtained "with a second step of distillation during fine-tuning". What does this involve exactly and how can it be performed with the DistilBERT model in this repo?
Hello @shreydesai,
You should have a look at [run_squad_w_distillation.py](https://github.com/huggingface/transformers/blob/master/examples/distillation/run_squad_w_distillation.py) which is the script used in the experiment you are mentioning.
Victor<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
