Couldn't reproduce DistilBERT downstream tasks performance on SQuAD dataset.
Hi, 

I tried the DIstilBERT finetuning (script path: transformers/examples/tensorflow/question-answering) for QuestionAnswering downstream task without changing any code, and this is my command:

`
python3 run_qa.py --model_name_or_path distilbert-base-uncased --output_dir output --dataset_name squad --do_train --do_eval
`

And finally, I got the following performance: SQuAD(EM/F1): 72.441/81.188

But the performance was shown on the paper: SQuAD(EM/F1): 79.1/86.9

Is there anything wrong with my command?

transformers version: 4.11.3
The `run_qa.py` script is very general, to be used by any `xxxForQuestionAnswering` model. To reproduce the results of a specific paper, you may need to tweak the hyperparameters, such as number of epochs, learning rate, batch size.<|||||>I see, thanks! @NielsRogge 

May I get the recommended hyperparameters to reproduce the performance of SQuAD(EM/F1): 79.1/86.9.<|||||>The DistilBERT author (@VictorSanh) used the (now legacy) `run_squad.py` script which can be found [here](https://github.com/huggingface/transformers/tree/master/examples/legacy/question-answering). So you might take a look at the hyperparameters used there. The following command is still included in the README of the question-answering examples:

```
python run_squad.py \
  --model_type distilbert \
  --model_name_or_path distilbert-base-uncased \
  --do_train \
  --do_eval \
  --do_lower_case \
  --train_file $SQUAD_DIR/train-v1.1.json \
  --predict_file $SQUAD_DIR/dev-v1.1.json \
  --per_gpu_train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 2.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir /tmp/debug_squad/
```
So might try this out. The checkpoint is also available on the hub: https://huggingface.co/distilbert-base-cased-distilled-squad<|||||>@NielsRogge Thanks so much!<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.