Issue with HANS evaluation
# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Albert, Bert

Language I am using the model on (English, Chinese ...): GLUE (MNLI), HANS

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce
I have tried to evaluate ALBERT and BERT (pretrained, trained on MNLI) on HANS. I'm using the official example. 

Everytime I run evaluation, I'm getting
```
Heuristic entailed results:
lexical_overlap: 0.0
subsequence: 0.0
constituent: 0.0

Heuristic non-entailed results:
lexical_overlap: 1.0
subsequence: 1.0
constituent: 1.0
```
Results can be reproduced by:
```
python3 hans/test_hans.py --task_name hans --model_type bert/albert --do_eval --data_dir $HANS_DIR --model_name_or_path $MODEL_PATH --max_seq_length 128 --output_dir $MODEL_PATH --per_gpu_eval_batch_size 1024 --overwrite_cache
```


## Expected behavior
It shouldn't be exactly 0 and 1 always and entailment score should be higher than non entailment.

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.11.0
- Platform: Linux 5.4.0-29-generic
- Python version: 3.8.2
- PyTorch version (GPU?): 1.5.0+cu101
- Tensorflow version (GPU?): NA
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

Could you please provide some possible areas to look at ? I can look at it and send a PR.<|||||>@sgugger @julien-c Thank you for the [PR](https://github.com/huggingface/transformers/issues/4742). There is a problem associated with this PR, that is, if you run `python3 evaluate_heur_output.py /path_to_hans_predictions.txt`, I'm getting an error:
```
guess = guess_dict[key]
Keyerror: 'ex0'
```
which suggests that a key is missing, indicating that `hans_predictions.txt` is not being generated in the expected manner as [HANS repo](https://github.com/huggingface/transformers/issues/4742) indicates. 
For fixing, it would great if evaluation is carried within transformers just like GLUE tasks and that user doesn't have to rely on external [repo](https://github.com/huggingface/transformers/issues/4742) to get predictions.  
It would be good if you can check this, as this error makes the example unusable.<|||||>I forgot the header on that file, #5082 should fix.<|||||>Evaluation seems to be working fine now. Feel free to close this issue @sgugger . Thanks again. Would be great if evaluation method can be integrated.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
