`QuestionAnsweringPipeline`: different behaviors when with `handle_impossible_answer=True`
### System Info

- `transformers` version: 4.1.1
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.9.1
- PyTorch version (GPU?): 1.7.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I am trying to use a Question Answering model on a question-context pair which is unanswerable. I would expect `""` (empty string) as the predicted output answer.

I am calling a `QuestionAnsweringPipeline` object on this sample with `handle_impossible_answer=True`, as recommended in the docs.

I tested two different models for Question Answering, `deepset/roberta-base-squad2` and `deepset/minilm-uncased-squad2`.
```python
import torch
from transformers import AutoTokenizer, AutoModelForQuestionAnswering, pipeline

question = "What is the capital of Belarus?"
context = "It rained yesterday."


for model_ckpt in ["deepset/minilm-uncased-squad2", "deepset/roberta-base-squad2"]:
    tokenizer = AutoTokenizer.from_pretrained(model_ckpt)
    model = AutoModelForQuestionAnswering.from_pretrained(model_ckpt)
    pipe = pipeline("question-answering", model=model, tokenizer=tokenizer)

    inputs = tokenizer(question, context, return_tensors="pt")
    with torch.no_grad():
        outputs = model(**inputs)

    pipe_answer = pipe(question=question, context=context, handle_impossible_answer=True)
        
    print(f"-- Model: {model_ckpt}")
    print(f"-- answer: {pipe_answer}")
    

    print(f"TOKEN           START   END")
    print("-----------------------------")
    for tok, st_, en_ in zip([tokenizer.decode(i) for i in inputs.input_ids[0]], outputs.start_logits[0], outputs.end_logits[0]):
        print(f"{tok:<15s} {st_:+.1f}   {en_:+.1f}")
        
    print()
```

The output I obtained is the following.
```text
-- Model: deepset/minilm-uncased-squad2
-- answer: {'score': 0.99960857629776, 'start': 0, 'end': 0, 'answer': ''}
TOKEN           START   END
-----------------------------
[CLS]           +4.7   +4.8
what            -5.2   -4.8
is              -5.8   -5.8
the             -4.5   -5.6
capital         -5.6   -5.8
of              -5.6   -6.2
belarus         -1.6   -2.0
?               -6.6   -5.6
[SEP]           +4.7   +4.8
it              -5.4   -6.2
rained          -5.9   -5.6
yesterday       -5.5   -4.8
.               -6.1   -3.9
[SEP]           +4.7   +4.8

-- Model: deepset/roberta-base-squad2
-- answer: {'score': 0.37058913707733154, 'start': 0, 'end': 20, 'answer': 'It rained yesterday.'}
TOKEN           START   END
-----------------------------
<s>             +1.1   +1.5
What            -7.1   -7.4
 is             -9.1   -8.0
 the            -8.3   -8.8
 capital        -7.9   -6.5
 of             -8.9   -8.2
 Belarus        -6.3   -5.0
?               -9.4   -7.7
</s>            -9.3   -8.3
</s>            -9.4   -8.3
It              -6.8   -8.6
 r              -7.5   -8.6
ained           -8.5   -6.9
 yesterday      -7.9   -6.7
.               -7.6   -3.6
</s>            -9.0   -8.5
```

### Expected behavior

Both models give very low scores to each token in the context for both `start` and `end` of the span, and since I passed `handle_impossible_answer=True` I would therefore expect the predicted output answer to be `""` (unanswerable) in both cases.

However (see output above) this is only true for `deepset/minilm-uncased-squad2`, while `deepset/roberta-base-squad2`surprisingly gives a non-empty answer.
Note: the bug is referred to `transformers 4.1.1`, but on the latest release `4.21.1` (thanks @EmileDel for noticing this!) the behavior seems consistent (see below). I don't know what has changed, but maybe this Issue can be closed?

```text
-- Model: deepset/minilm-uncased-squad2
-- answer: {'score': 0.9996085166931152, 'start': 0, 'end': 0, 'answer': ''}
TOKEN           START   END
-----------------------------
[CLS]           +4.7   +4.8
what            -5.2   -4.8
is              -5.8   -5.8
the             -4.5   -5.6
capital         -5.6   -5.8
of              -5.6   -6.2
belarus         -1.6   -2.0
?               -6.6   -5.6
[SEP]           +4.7   +4.8
it              -5.4   -6.2
rained          -5.9   -5.6
yesterday       -5.5   -4.8
.               -6.1   -3.9
[SEP]           +4.7   +4.8

-- Model: deepset/roberta-base-squad2
-- answer: {'score': 0.9927088618278503, 'start': 0, 'end': 0, 'answer': ''}
TOKEN           START   END
-----------------------------
<s>             +1.1   +1.5
What            -7.1   -7.4
 is             -9.1   -8.0
 the            -8.3   -8.8
 capital        -7.9   -6.5
 of             -8.9   -8.2
 Belarus        -6.3   -5.0
?               -9.4   -7.7
</s>            -9.3   -8.3
</s>            -9.4   -8.3
It              -6.8   -8.6
 r              -7.5   -8.6
ained           -8.5   -6.9
 yesterday      -7.9   -6.7
.               -7.6   -3.6
</s>            -9.0   -8.5
``` <|||||>cc @Narsil <|||||>Hi @FrancescoCasalegno ,

I can't find the PR associated with this, but yes, early on the pipelines were roughly hardcoded and sometimes wouldn't work on some models because of differences in tokenizers for instance. I won't say everything is fixed, but we caught a bunch of these, and now behavior should be more consistent across models so hopefully you can easily swap out models and get a consistent behavior even when the underlying models are quite different.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.