how to get distilbert-base-uncased-distilled-squad?
## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I think it is initialize a six layer bert and distill with a 12 layer bert, then save the checkpoint file, is it right?
Hello @RyanHuangNLP,
I am not sure to get your question.
Do you mean that you want to play with the model (do inferences)? If so, did you try `qa_model = DistilBertForQuestionAnswering.from_pretrained('distilbert-base-uncased-distilled-squad')`?<|||||>@VictorSanh I wonder how to get the 'distilbert-base-uncased-distilled-squad' pretrain model, just use the first six layer of the base one or initialize a six layer bert?<|||||>If you do:
```
qa_model = DistilBertForQuestionAnswering.from_pretrained('distilbert-base-uncased-distilled-squad')
```
You will get the 'distilbert-base-uncased-distilled-squad' pretrain model.
Nothing more to do.<|||||>I am sorry not clearly expressed, my question is how to get the 'distilbert-base-uncased-distilled-squad' pretrain model,  I know that use that code can get the six layer layer, but how to train the pretrain model is what I concern, there is no six layer bert release<|||||>Ok, I understand your question now @RyanHuangNLP.
It is finetuned from `distilbert-base-uncased`. More precisely, the model we release is finetuned AND distilled at the same time: the loss is computed from the classic qa loss (see `run_squad.py` and `DistilBertForQuestionAnswering`) plus the distillation supervision from a BERT SQuAD model (second loss).
We haven't released the script for doing that (and we plan to do it in the near future) but it is a simple adaptation of `run_squad.py` (mostly adding a second loss i.e. distillation).<|||||>do we have any release date of that run_squad_adapted.py @VictorSanh ? :D<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
