wav2vec2: adding single-char tokens to tokenizer causes tokenization mistakes
## Environment info

- `transformers` version: 4.4.0.dev0
- Platform: Linux-5.8.0-44-generic-x86_64-with-glibc2.10
- Python version: 3.8.8
- PyTorch version (GPU?): 1.8.0 (True)
- Tensorflow version (GPU?): 2.4.1 (False)
- Using GPU in script?: N/A
- Using distributed or parallel set-up in script?: N/A

### Who can help

@patrickvonplaten and @LysandreJik

Issue is probably related to interactions of the following:

https://github.com/huggingface/transformers/blob/9a8c168f56fe3c0e21d554a577ac03beb004ef89/src/transformers/tokenization_utils.py#L213

https://github.com/huggingface/transformers/blob/11fdde02719dbd20651c9f43cc6f54959fc6ede6/src/transformers/tokenization_utils.py#L352

https://github.com/huggingface/transformers/blob/cb38ffcc5e0ae2fac653342ac36dc75c15ea178f/src/transformers/models/wav2vec2/tokenization_wav2vec2.py#L184

This is a corner case: `add_tokens` adds new tokens to `self.unique_no_split_tokens` -- causing `tokenize()` to skip calling `Wav2Vec2CTCTokenizer._tokenize()`

This is probably not the case with most tokenizers since their vocab includes most, if not all, commonly used single-characters tokens without including them in `self.unique_no_split_tokens`. I faced this while debugging my code for https://github.com/huggingface/transformers/pull/10581 to add support for Buckwalter Arabic transliteration. The issue is not limited to adding single-char tokens but rather when words (space-separated) start or end with a newly added token.

## Information

Model I am using (Bert, XLNet ...): wav2vec2

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: adding tokens to ASR vocab

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: training an ASR with extended vocab

## To reproduce

Steps to reproduce the behavior:

```python
from transformers import Wav2Vec2Processor

tokenizer = Wav2Vec2Processor.from_pretrained('facebook/wav2vec2-base').tokenizer
tokenizer.add_tokens('x')
token_ids = tokenizer('C x A').input_ids
decoded = tokenizer.decode(token_ids)
print(decoded, token_ids)
# CxA [19, 32, 7]
```

## Expected behavior

Should have printed `C x A [19, 4, 32, 4, 7]`

My workaround right now is to keep a reference to the original `tokenizer.unique_no_split_tokens` before adding tokens then restoring it afterwards:

```python
from transformers import Wav2Vec2Processor

tokenizer = Wav2Vec2Processor.from_pretrained('facebook/wav2vec2-base').tokenizer
unique_no_split_tokens = tokenizer.unique_no_split_tokens
tokenizer.add_tokens('x')
tokenizer.unique_no_split_tokens = unique_no_split_tokens
token_ids = tokenizer('C x A').input_ids
decoded = tokenizer.decode(token_ids)
print(decoded, token_ids)
# C x A [19, 4, 32, 4, 7]
```<|||||>Hey @elgeish,

Sorry for replying that late! Yes, you are absolutely right here :-)
I think we should overwrite the `add_tokens(self, ...)` function in 
`src/transformers/models/wav2vec2/tokenization_wav2vec2.py` with the "hack" just as you did:

```python
def _add_tokens(self, new_tokens: Union[List[str], List[AddedToken]], special_tokens: bool = False) -> int:
   # copy past the function from `src/transformers/tokenization_utils.py` 
   # + add the "hack": 
   unique_no_split_tokens = tokenizer.unique_no_split_tokens
   tokenizer.unique_no_split_tokens = unique_no_split_tokens
```

If you want and have some time, it would be amazing if you could open a PR :-)
<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>Hey @patrickvonplaten,
Shall I open a PR for this issue.<|||||>Hey @Muktan,

yes this would be great :-)