KeyError in Camembert in QuestionAnsweringPipeline
# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):
`Camembert ("illuin/camembert-large-fquad")`
Language I am using the model on (English, Chinese ...):
`French`
The problem arises when using:
* [X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [X] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce
Load model and create a question answering pipeline
Steps to reproduce the behavior:

1. 
```
from transformers import (QuestionAnsweringPipeline, CamembertForQuestionAnswering, CamembertModel, CamembertTokenizer)
```
2. 
```
QA_model = "illuin/camembert-large-fquad"
CamTokQA = CamembertTokenizer.from_pretrained(QA_model)
CamQA = CamembertForQuestionAnswering.from_pretrained(QA_model)
```
3.
```
device_pipeline = 0 if torch.cuda.is_available() else -1
q_a_pipeline = QuestionAnsweringPipeline(model=CamQA,
                                         tokenizer=CamTokQA,
                                         device=device_pipeline)
```
4. 
```
res = q_a_pipeline({'question': question, 'context': ctx})
```
<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

```
  File "/mnt/Documents/Projets/BotPress/R_D/R_D_q_a/sdk/readers.py", line 15, in get_answers
    res = q_a_pipeline({'question': question, 'context': ctx})
  File "/home/pedro/.local/lib/python3.8/site-packages/transformers/pipelines.py", line 1213, in __call__
    answers += [
  File "/home/pedro/.local/lib/python3.8/site-packages/transformers/pipelines.py", line 1216, in <listcomp>
    "start": np.where(char_to_word == feature.token_to_orig_map[s])[0][0].item(),
KeyError: 339
```

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
Get answer from the Q_a pipeline

*Is working on transformers version 2.8.0* (but not on next ones)

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.10.0
- Platform: Linux-5.6.14-arch1-1-x86_64-with-glibc2.2.5
- Python version: 3.8.3
- PyTorch version (GPU?): 1.5.0 (True)
- Tensorflow version (GPU?): 2.2.0-rc4 (False)
- Using GPU in script?: Yes (but same problem with cpu only)
- Using distributed or parallel set-up in script?: No

I can't reproduce on master. Do you mind specifying the context/question?<|||||>Because you couldn't reproduce, I tried with the latest version (from git : 2.11.0) and the problem seems gone (tested with random articles from wikipedia).  

Maybe my first text had bad format or non utf-8 characters (but I remember to have tested with many differents inputs before openning an issue) or it was due to a bug fixed in 2.11

Sorry to have bothered you, thanks for the support ! 
<|||||>Alright, no worries! Let me know if you have an issue down the road.<|||||>@LysandreJik 
I got the exact same error using Camembert ("illuin/camembert-large-fquad") and the question answering pipeline.

Opening a new issue.

Questions :
Le loyer est-il révisé annuellement ou triennalemment ?
Quel est la nature de l’indice de base ?
Le bail est-il soumis à TVA ou non soumis à TVA ?

Context : 
[context_mono.txt](https://github.com/huggingface/transformers/files/4752269/context_mono.txt)

```
Traceback (most recent call last):
  File "qa.py", line 164, in <module>
    main_file()
  File "qa.py", line 161, in main_file
    analayse(mode)
  File "qa.py", line 85, in analayse
    answer_C = nlp_camembert_gpu_f(question=question_text, context=context)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/transformers/pipelines.py", line 1229, in __call__
    for s, e, score in zip(starts, ends, scores)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/transformers/pipelines.py", line 1229, in <listcomp>
    for s, e, score in zip(starts, ends, scores)
KeyError: 377
```