`run_squad.py` not using the dev cache
## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): Bert (but it's independent of the model)

Language I am using the model on (English, Chinese....): English (but it's independent of the language)

The problem arise when using:
* [ X] the official example scripts: `examples/run_squad.py`

The tasks I am working on is:
* [ X] an official GLUE/SQUaD task: (give the name)

It's not really a bug, more an unnecessary repetition of some operations.

It seems like the dev set is binarized (tokenized + tokens_to_id) for every single evaluation of a checkpoint even if the binarized data are already cached (from the previous evaluation for instance). It is particularly striking when adding the flag `--eval_all_checkpoints`.

It arises when calling `dataset, examples, features = load_and_cache_examples(args, tokenizer, evaluate=True, output_examples=True)` in the `evaluate` function. The cache is never used because of the argument `output_examples=True`:

```python
if os.path.exists(cached_features_file) and not args.overwrite_cache and not output_examples:
    logger.info("Loading features from cached file %s", cached_features_file)
    features = torch.load(cached_features_file)
```
From my understanding, except if the tokenizer changes between two checkpoints (which is not the case), the computed features are always the same.

The command I use:

```bash
python -m torch.distributed.launch --nproc_per_node=8 ./examples/run_squad.py \
    --model_type bert \
    --model_name_or_path bert-base-uncased \
    --do_train \
    --do_eval \
    --do_lower_case \
    --train_file $SQUAD_DIR/train-v1.1.json \
    --predict_file $SQUAD_DIR/dev-v1.1.json \
    --learning_rate 3e-5 \
    --num_train_epochs 2 \
    --max_seq_length 384 \
    --doc_stride 128 \
    --output_dir ../models/bert-base-uncased_finetuned_squad/ \
    --per_gpu_eval_batch_size=3   \
    --per_gpu_train_batch_size=3   \
    --eval_all_checkpoints 
```
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
