[Tokenizers] Missmatch between fast and slow 
When I worked on the implementation of Whisper I realised that two different behaviors appear when you use a `fast` or `slow` tokenizer and have a OOV.
Simple snippet : 
```python
>>> from transformers import GPT2Tokenizer, GPT2TokenizerFast
>>> fast = GPT2TokenizerFast.from_pretrained("gpt2")
>>> slow = GPT2Tokenizer.from_pretrained("gpt2")
>>> # the vocab size is 50257
>>> fast.decode(50258)
''
```
```python 
>>> slow.decode(50258)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/arthur_huggingface_co/transformers/src/transformers/tokenization_utils_base.py", line 3468, in decode
    return self._decode(
  File "/home/arthur_huggingface_co/transformers/src/transformers/tokenization_utils.py", line 938, in _decode
    for token in filtered_tokens:
TypeError: 'NoneType' object is not iterable
```

My question I guess is : which one is the expected one?
Here is my take : 
- It should work, but output a warning saying that an OOV was encountered and was ignored. 

WDYT @sgugger @LysandreJik @Narsil  
Thanks for reporting.

- `tokenizers` CAN do warnings (although I was under the impression there was an effort to reduce warnings.).
- In my personal opinion, raising an Exception is better when OOV than silently ignoring. That being said, it would be a massive breaking change, so I'm hesitant to "fix" that way.<|||||>Yes, an exception should be raised and it's more of a bug fix than a breaking change IMO. Users will be surprised, but they should be surprised when there is an out-of-vocab index.<|||||>@Narsil @sgugger  I can take this up. Will raise an exception in PreTrainedTokenizer when OOV is encountered . <|||||>(This might break the tokenizer used by Whisper, as all of the timestamp tokens are not `in` the vocabulary, but are still used and need to be decoded as `''`)<|||||>Couldn't we put them in the vocab as `Timestamp <n>` (being the seconds offset ?) (Even as special tokens ?)<|||||>we can, but we would also have to help the openAi team with their tokenizer that is based on pour GPT2TokenizerFast 😅<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.