Issue with long contexts or AutoTokenizer.. Unsure which one is it is more of
Having an issue where I used a QA model from transformers, tweaked the model in label studio making some annotations and then tried to load the model back again. The model is pulling out the correct answers but seemingly `handle_impossible_answers` isn't working because it gives an answer for every questions even when the question is irrelevant.. What's even weirder is that it doesn't do this in label studio's interface so seemingly this handle_impossible_answers is working on that side. My contexts I pass through the pipeline are also quite long

Made sure I have the most up to date transformers model, and the models were saved out with save_pretrained. 
```
# Inside some function code

model_to_save = model.module if hasattr(model, "module") else model  # Take care of distributed/parallel training  # noqa
# If we save using the predefined names, we can load using `from_pretrained`
model_to_save.save_pretrained(workdir)
self.tokenizer.save_pretrained(workdir)
```

And I end up seeing all the relevant files in a local dir as follows:
```
# In the model directory

job_result.json    special_tokens_map.json  tokenizer.json        train_data.json
config.json     pytorch_model.bin  tokenizer_config.json    train_data_info.json  vocab.txt
```

I then try and load that model using the exact same transformers model/tokenizer in the way I did before, except swapping it out for the local files. (Also read some bug about local path vs absolute paths when loading models and that made no difference either way for me)

```
model_path = "/some/path/to/model_dir/"
model = AutoModelForQuestionAnswering.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path) 
QA = pipeline('question-answering', model=model, tokenizer=tokenizer)

# Other code...
#....

answers = QA(question=questions, context=context, top_k=1, max_answer_len=32, handle_impossible_answer=True)
```

The answers that I get out of here have an answer for every question, even when it doesn't need to be answered. I've tried loading the models multiple ways - the only way I got handle_impossible_answers to work as intended is if the tokenizer wasn't the AutoTokenizer that I had used here (swapped it out for Bert). But then the answers it gave me were complete garbage which was kind of expected. Anybody else run into this issue with AutoTokenizer??

If the context is short (most of the contexts I hit it with are fairly long) then it actually does do a good job of only pulling out the valid answers. So I'm not sure if this is a long context issue, a problem with AutoTokenizer, or even a problem with transformers.pipeline