the results of run_squad.py is terrible
# ❓ the results of run_squad.py is terrible
## I runned run_squad.py and the results is as folllows:
`Results: {'exact': 40.25098964036048, 'f1': 44.224848486777795, 'total': 11873, 'HasAns_exact': 80.61740890688259, 'HasAns_f1': 88.57652261867625, 'HasAns_total': 5928, 'NoAns_exact': 0.0, 'NoAns_f1': 0.0, 'NoAns_total': 5945, 'best_exact': 50.11370336056599, 'best_exact_thresh': 0.0, 'best_f1': 50.11370336056599, 'best_f1_thresh': 0.0}`

_**`exact`,`f1` and `HasAns_exact`,`HasAns_f1` are very different!**_

## The startup args are as follows:

export SQUAD_DIR=./datasets/squad

python run_squad.py \
  --squad_model bert_qa_model_squad \
  --model_type bert \
  --model_name_or_path bert-base-uncased \
  --do_train \
  --do_eval \
  --do_lower_case \
  --train_file $SQUAD_DIR/train-v2.0.json \
  --predict_file $SQUAD_DIR/dev-v2.0.json \
  --per_gpu_train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 2.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir ./models/debug_squad/


Maybe I not use the argument `--version_2_with_negative` while training on Squad V2.0 ?<|||||>Indeed, not using that clarg would result in such bad results as you're not taking into account impossible answers. Can you let us know if you obtain better results with `--version_2_with_negative`?<|||||>> Indeed, not using that clarg would result in such bad results as you're not taking into account impossible answers. Can you let us know if you obtain better results with `--version_2_with_negative`?

After using `--version_2_with_negative`, the result is also not satisfactory.
Both exact and f1 are not exceed **80%**.

The results are as follows:
{"exact_20000": 73.88191695443443, "f1_20000": 76.9165287314905, "total_20000": 11873, "HasAns_exact_20000": 68.60661268556005, "HasAns_f1_20000": 74.68453873633385, "HasAns_total_20000": 5928, "NoAns_exact_20000": 79.1421362489487, "NoAns_f1_20000": 79.1421362489487, "NoAns_total_20000": 5945, "best_exact_20000": 73.88191695443443, "best_exact_thresh_20000": 0.0, "best_f1_20000": 76.91652873149043,}

After the 25000th steps of training, the performance got worse.
<|||||>That's already a lot better! What is this clarg `--squad_model bert_qa_model_squad`? It's not in the `run_squad.py` script?<|||||>


> That's already a lot better! What is this clarg `--squad_model bert_qa_model_squad`? It's not in the `run_squad.py` script?

Oh ,I have adjusted some code from `run_squad.py` so that it could load and train my custom model from clarg.
And `bert_qa_model_squad` is the model whose code is copied from `BertForQuestionAnswering` in `transformers`.
So that I can design my custom model and then transmit it to the `run_squad.py` script.<|||||>Does your script differ in any other way to the `run_squad.py` script? Does your model differ in any way to the `BertForQuestionAnswering` model in `transformers`? Have you tried running the standard command on the `transformers` script and model?

```
python run_squad.py
--model_name_or_path bert-base-uncased
--do_train
--do_eval
--do_lower_case
--train_file $SQUAD_DIR/train-v2.0.json
--predict_file $SQUAD_DIR/dev-v2.0.json
--version_2_with_negative
--per_gpu_train_batch_size 12
--learning_rate 3e-5
--num_train_epochs 2.0
--max_seq_length 384
--doc_stride 128
--output_dir ./models/debug_squad/
```<|||||>> Does your script differ in any other way to the `run_squad.py` script? Does your model differ in any way to the `BertForQuestionAnswering` model in `transformers`? Have you tried running the standard command on the `transformers` script and model?
> 
> ```
> python run_squad.py
> --model_name_or_path bert-base-uncased
> --do_train
> --do_eval
> --do_lower_case
> --train_file $SQUAD_DIR/train-v2.0.json
> --predict_file $SQUAD_DIR/dev-v2.0.json
> --version_2_with_negative
> --per_gpu_train_batch_size 12
> --learning_rate 3e-5
> --num_train_epochs 2.0
> --max_seq_length 384
> --doc_stride 128
> --output_dir ./models/debug_squad/
> ```

sorry, I will try later and tell you the results.<|||||>> Does your script differ in any other way to the `run_squad.py` script? Does your model differ in any way to the `BertForQuestionAnswering` model in `transformers`? Have you tried running the standard command on the `transformers` script and model?
> 
> ```
> python run_squad.py
> --model_name_or_path bert-base-uncased
> --do_train
> --do_eval
> --do_lower_case
> --train_file $SQUAD_DIR/train-v2.0.json
> --predict_file $SQUAD_DIR/dev-v2.0.json
> --version_2_with_negative
> --per_gpu_train_batch_size 12
> --learning_rate 3e-5
> --num_train_epochs 2.0
> --max_seq_length 384
> --doc_stride 128
> --output_dir ./models/debug_squad/
> ```
I have run the official `run_squad.py` script，and the result is as follows:
10/16/2020 19:34:00 - INFO - main -   Results: {'exact': 72.60170133917292, 'f1': 75.79599520268259, 'total': 11873, 'HasAns_exact': 72.2165991902834, 'HasAns_f1': 78.61434734167507, 'HasAns_total': 5928, 'NoAns_exact': 72.9857022708158, 'NoAns_f1': 72.9857022708158, 'NoAns_total': 5945, 'best_exact': 72.60170133917292, 'best_exact_thresh': 0.0, 'best_f1': 75.79599520268256, 'best_f1_thresh': 0.0}
<|||||>i have ran the official run_squad.py script, but got a 'cached_train_bert-base-uncased_384' not a result. Did i something wrong?<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
