Fast tokenizers calculate wrong offsets when special characters are present
Example:

```
>>> import transformers
>>> t_fast = transformers.AutoTokenizer.from_pretrained("bert-base-cased", use_fast=True, add_special_tokens=False)
>>> sentence = "A, naïve [MASK] AllenNLP sentence."
>>> tokenized = t_fast.encode_plus(sentence, add_special_tokens=False, return_offsets_mapping=True)
>>> for start, end in tokenized['offset_mapping']:
...     print(repr(sentence[start:end]))
'A'
','
'naïve'
' [MASK'
' Alle'
'nN'
'L'
' sentenc'
'e'
```

As you can see, after the word "naïve", the offsets go off the rails.
@mfuntowicz, would it make sense for me to integrate our tokenizer tests into your code, so you can see these things immediately? I'd be happy to do so.<|||||>Looks like this is a duplicate of #2917.<|||||>On second reading, this is not the same issue as #2917, though they may be related.<|||||>Roberta has another related issue:
```
>>> import transformers
>>> t_fast = transformers.AutoTokenizer.from_pretrained("roberta-base", use_fast=True, add_special_tokens=False)
>>> sentence = "I went to the zoo yesterday, but they had only one animal."
>>> tokenized = t_fast.encode_plus(sentence, add_special_tokens=False, return_offsets_mapping=True)
>>> for start, end in (t for t in tokenized['offset_mapping'] if t is not None):
...     print(repr(sentence[start:end]))
'I'
' went'
' to'
' the'
' zoo'
' yesterday'
','
' but'
' they'
' had'
' only'
' one'
' animal'
'.'
```

There are two problems here. `add_special_tokens` is being ignored (#2919), but also, it adds those extra spaces at the front of the words.<|||||>Hi @dirkgr, 

Thanks for your report.

Regarding the integration of your tests, it definitively a good idea, if you can put @LysandreJik and myself as reviewers of the PR, we'll have a look 👍.

Regarding `add_special_tokens`, the behaviour on Rust side is quite different as it's a parameter that needs to be provided a construction time, whereas Python allows at tokenisation time. We should make it clearer in the doc.

```python
>>> t = transformers.BertTokenizerFast.from_pretrained('bert-base-cased')
>>> tokenized = t.encode_plus(sentence, add_special_tokens=False, return_offsets_mapping=True)
>>> tokenized['input_ids']
[101, 138, 117, 22607, 103, 4522, 20734, 2101, 5650, 119, 102]

>>> t = transformers.BertTokenizerFast.from_pretrained('bert-base-cased', add_special_tokens=False)
>>> tokenized = t.encode_plus(sentence, add_special_tokens=False, return_offsets_mapping=True)
>>> tokenized['input_ids']
>>> [138, 117, 22607, 103, 4522, 20734, 2101, 5650, 119]
```

For Roberta we're aware of this extra space being included, cc'ing @n1t0 for keeping track of this.<|||||>Ok some more context, `GPT2TokenizerFast` has also the same behaviour, so I would extrapolate this occurs more generally on BPE model. 

Adding `<ModelFast>.from_pretrained(..., add_prefix_space=True)` doesn't append the space before but after the token:

```python
'I '
'went '
'to '
'the '
'zoo '
'yesterday,'
' '
'but '
'they '
'had '
'only '
'one '
'animal.'
```<|||||>Hi @dirkgr!

So there are multiple different things here:
 - Your first example is due to the space between `naïve` and `[MASK]` not being trimmed out. We are aware of this behavior and are currently working on a fix.
 - The fact that `add_prefix_space=True` moves the space at the end is actually a bug too. This happens because we mess with the offsets while adding the prefix. I am working on a fix for this too.
 - Now, the second example you gave is actually expected behavior:
```python
import transformers
t_fast = transformers.AutoTokenizer.from_pretrained("roberta-base", use_fast=True, add_special_tokens=False)
sentence = "I went to the zoo yesterday, but they had only one animal."
tokenized = t_fast.encode_plus(sentence, return_offsets_mapping=True)
offsets = tokenized['offset_mapping']
tokens = t_fast.tokenize(sentence)
for token, (start, end) in (t for t in zip(tokens, offsets) if t[1] is not None):
    print(repr(token))
    print(repr(sentence[start:end]))
    print(repr(t_fast.decode(t_fast.convert_tokens_to_ids([token]))))
```
will give the following output:
```
'I'
'I'
'I'
'Ġwent'
' went'
' went'
'Ġto'
' to'
' to'
'Ġthe'
' the'
' the'
'Ġzoo'
' zoo'
' zoo'
'Ġyesterday'
' yesterday'
' yesterday'
','
','
','
'Ġbut'
' but'
' but'
'Ġthey'
' they'
' they'
'Ġhad'
' had'
' had'
'Ġonly'
' only'
' only'
'Ġone'
' one'
' one'
'Ġanimal'
' animal'
' animal'
'.'
'.'
'.'
```
Here you can see that the space is actually part of these tokens. That's just the way the byte-level BPE used by GPT-2 and Roberta works. The `Ġ` is actually an encoded space. Does it make sense?<|||||>We hit a similar issue when we add new tokens. The input ids are correct, but offsets after the new token are off.

Example: https://colab.research.google.com/drive/1e2a3iyLF9NSMWZR50pnDYRhpizcsRV_6

```
text = "A [test] C"

print(tokenizer.encode(text, add_special_tokens=True))

results = tokenizer.encode_plus(text, 
            return_offsets_mapping=True,
            pad_to_max_length=False,
            max_length=128,
            return_overflowing_tokens=False,
            add_special_tokens=True)

for se in results['offset_mapping']:
  if se:
    print(text[se[0]:se[1]], se)
```

```
[101, 1037, 30522, 1039, 102]
A (0, 1)
 [test (1, 7)
  (8, 9)
```

Potentially related issue huggingface/tokenizers#143
<|||||>I think all of the mentioned bugs on this issue should now be fixed on `master`<|||||>```
>>> import transformers
>>> t_fast = transformers.AutoTokenizer.from_pretrained("bert-base-cased", use_fast=True, add_special_tokens=False)
>>> sentence = "A, naïve [MASK] AllenNLP sentence."
>>> tokenized = t_fast.encode_plus(sentence, add_special_tokens=False, return_offsets_mapping=True)
>>> for start, end in tokenized['offset_mapping']:
...     print(repr(sentence[start:end]))
'A'
','
'naïve'
'[MASK]'
'Allen'
'NL'
'P'
'sentence'
'.'
```
and
```
>>> import transformers
>>> t_fast = transformers.AutoTokenizer.from_pretrained("roberta-base", use_fast=True, add_special_tokens=False)
>>> sentence = "I went to the zoo yesterday, but they had only one animal."
>>> tokenized = t_fast.encode_plus(sentence, add_special_tokens=False, return_offsets_mapping=True)
>>> for start, end in (t for t in tokenized['offset_mapping'] if t is not None):
...     print(repr(sentence[start:end]))
'I'
'went'
'to'
'the'
'zoo'
'yesterday'
','
'but'
'they'
'had'
'only'
'one'
'animal'
'.' 
```
and the last one:
```
from transformers import BertTokenizerFast
tokenizer = BertTokenizerFast.from_pretrained('bert-base-uncased')
tokenizer.add_tokens(['[test]'])
text = "A [test] C"

print(tokenizer.encode(text, add_special_tokens=True))

results = tokenizer.encode_plus(text, 
            return_offsets_mapping=True,
            pad_to_max_length=False,
            max_length=128,
            return_overflowing_tokens=False,
            add_special_tokens=True)

for se in results['offset_mapping']:
  if se:
    print(text[se[0]:se[1]], se)
```
gives
```
[101, 1037, 30522, 1039, 102]
 (0, 0)
A (0, 1)
[test] (2, 8)
C (9, 10)
 (0, 0)
```