GPT2Tokenizer strips spaces surrounding special tokens
## Environment info
     
- `transformers` version: 3.3.1
- Platform: Darwin-19.6.0-x86_64-i386-64bit
- Python version: 3.7.9
- PyTorch version (GPU?): 1.6.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help

@mfuntowicz, this seems to be an issue related to tokenization. So I hope you are the right person to ping here. 

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Minimal working example:

```py
import os
os.makedirs('models/', exist_ok=True)

from transformers import GPT2Tokenizer
from tokenizers import ByteLevelBPETokenizer


open('train.txt', 'w').write('Training data including a <special> token.')

special_tokens = ['<special>']

bpe_tokenizer = ByteLevelBPETokenizer()
bpe_tokenizer.train(
    files=['train.txt'],
    special_tokens=special_tokens
)
bpe_tokenizer.save_model('models/')

gpt2_tokenizer = GPT2Tokenizer.from_pretrained(
    'models/',
    additional_special_tokens=special_tokens,
)
```

When encoding below text, the two tokenizers yield different outputs:

```py
>>> text = 'A <special> token.'
>>> bpe_tokenizer.encode(text).tokens
['A', 'Ġ', '<special>', 'Ġ', 't', 'o', 'k', 'e', 'n', '.']
>>> gpt2_tokenizer.tokenize(text)
['A', '<special>', 't', 'o', 'k', 'e', 'n', '.'] # <----- Note the missing space (`Ġ`) around `<special>` 
```

## Expected behavior

I would expect that both tokenizers give the same output when encoding the sentence. Furthermore, because `GPT2Tokenizer` seems to remove the spaces surrounding the special token, the decode(encode()) does not return the original string. 

```py
assert bpe_tokenizer.encode(text).tokens == ['A', 'Ġ', '<special>', 'Ġ', 't', 'o', 'k', 'e', 'n', '.']
assert gpt2_tokenizer.tokenize(text) == ['A', 'Ġ', '<special>', 'Ġ', 't', 'o', 'k', 'e', 'n', '.']
```

It is possible that I misunderstand the `GPT2Tokenizer` API. Please advise if I should pass `special_tokens` in a different way. Thank you in advance.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>The issue is still present.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>This issue is still present in `transformers==4.5.1`.<|||||>Hey @jantrienes, sorry for getting back so late to you. The issue here is on our side - the added tokens work differently between the slow and fast tokenizers and we'll have to patch that. In the meantime, you can use `AddedToken`s instead of the strings to define the strategy w.r.t the stripping of whitespace. This should behave normally:

```py
import os
os.makedirs('models/', exist_ok=True)

from transformers import GPT2Tokenizer
from tokenizers import ByteLevelBPETokenizer, AddedToken


open('train.txt', 'w').write('Training data including a <special> token.')

special_tokens = [AddedToken('<special>')]

bpe_tokenizer = ByteLevelBPETokenizer()
bpe_tokenizer.train(
    files=['train.txt'],
    special_tokens=special_tokens
)
bpe_tokenizer.save_model('models/')

gpt2_tokenizer = GPT2Tokenizer.from_pretrained(
    'models/',
)
gpt2_tokenizer.add_special_tokens({"additional_special_tokens": special_tokens})

text = 'A <special> token.'

print(bpe_tokenizer.encode(text).tokens)
print(gpt2_tokenizer.tokenize(text))
assert bpe_tokenizer.encode(text).tokens == ['A', 'Ġ', '<special>', 'Ġ', 't', 'o', 'k', 'e', 'n', '.']
assert gpt2_tokenizer.tokenize(text) == ['A', 'Ġ', '<special>', 'Ġ', 't', 'o', 'k', 'e', 'n', '.'
```

Note how the special tokens are defined using `AddedToken` instead of a string. Unfortunately, these cannot be passed during initialization as you've done, but I'm fixing this in https://github.com/huggingface/transformers/pull/11325.

You can control the `AddedToken`'s behavior relative to whitespace using the `rstrip` and `lstrip` keyword arguments.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>This issue is present for "non-special" tokens in transformer 4.12.3.  The code explicitly strips spaces from the surrounding tokens.  It does NOT look for the `AddedToken` class to get the behavior so there is no way to change this.

See `tokenization_utils.py::tokenizer()` line 517:
```
         else:
              # We strip left and right by default
              if right:
                  tokens[i + 1] = right.lstrip()
              if left:
                  tokens[i - 1] = left.rstrip()
```
This leads to incorrect behavior unless the added token is in the middle of a longer word.

I should point out that the situation is more complicated than just changing the behavior above.  Even if you comment out those lines (at least with the T5Tokenizer) there are still no spaces around the added tokens because, just below those lines, the logic to concatenate all the tokens is also stripping the spaces as well (see `tokenized_text.extend(self._tokenize(token))`).

I'm not sure what the right solution is but this is in the base class so it's happening for a number of tokenizers including T5 and bart.  It is also happening for T5Fast.<|||||>@SaulLu, would you like to take a look at this?<|||||>Glad to look at this issue in more detail! I'll dive in tomorrow :monocle_face: <|||||>
> I'm not sure what the right solution is but this is in the base class so it's happening for a number of tokenizers including T5 and bart. It is also happening for T5Fast.

Yeah, I'm noticing spaces around my added tokens when creating a custom Bart tokenizer. Did this get resolved for you? Is there a way to work around it? 