Issue in fine-tuning distilbert on Squad 1.0
When I tried to finetune the distillbert (using run_squad.py in examples folder), the model reaches the F1 score of 17.43 on dev set but you have mentioned that the F1 score is 86.2. Can you help me with what I am doing wrong at the time of fine-tuning?

Below is the command that I am using
python ./examples/run_squad.py \
    --model_type bert \
    --model_name_or_path /root/distilbert_training \
    --do_train \
    --do_eval \
    --do_lower_case \
    --train_file train-v1.1.json \
    --predict_file dev-v1.1.json \
    --learning_rate 3e-5 \
    --num_train_epochs 2 \
    --max_seq_length 384 \
    --doc_stride 128 \
    --output_dir /root/output/ \
    --per_gpu_eval_batch_size=3 \
    --per_gpu_train_batch_size=3
Hello @pragnakalpdev,
Did you change the `run_squad.py` file to include distilbert ([here](https://github.com/huggingface/pytorch-transformers/blob/master/examples/run_squad.py#L58) for instance)?
Can you check the warnings ``Weights from XXX not used in YYY``?<|||||>Hello @VictorSanh,
Thank You for your response. I got the F1 score 86.4.
But now when I am facing issue in the evaluation process.
I had performed the evaluation on my custom file that has a single paragraph and 5 questions, and its taking approx 10 seconds to generate the prediction.json file. 
I want the inference time about 1-2seconds, what can I do for that?<|||||>Hello @pragnakalpdev7,
Good!

How long is your paragraph? Is the inference time during the training/fine-tuning the same as during test? (a forward pass during training should be slightly slower).
If you're already using `run_squad.py`, there is no easy/direct way to accelerate the inference.<|||||>Hello @VictorSanh ,
My paragraph is less than 1000 characters, and yes i am already using the run_squad.py for inference.
and i didn't understand this - "Is the inference time during the training/fine-tuning the same as during test? (a forward pass during training should be slightly slower)."

For fine-tuning it took 4 hours on 1 GPU, and now i am using the fined-tuned model for inference which is taking 8 - 9 seconds.<|||||>I mean that if a forward pass already takes ~2 sec during training on your machine, it is not likely to go down to 0.1 sec during test. The reason why a forward pass is on average slightly faster during test is that the flag `torch.no_grad` deactivate the autograd engine.<|||||>Hi @pragnakalpdev , can I ask you how you solved the first problem to achieve good performance? I'm in a similar situation and any hints would help.<|||||>Hi @pragnakalpdev  could you indeed please comment how you solved the first problem (bad F1) to achieve good performance?<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>Hi, can anyone tell me where(exactly in which directory or in colab exactly where) to run that code to fine-tune the bert model? I can't able to fine-tune my bert model for QnA purposes. Please reply.