XLMRobertaTokenizer potential bug 
## Environment info

- `transformers` version: 3.4.0
- Platform: Linux-5.4.60-1-pve-x86_64-with-debian-buster-sid
- Python version: 3.6.3
- PyTorch version (GPU?): 1.7.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help
@LysandreJik 
@mfuntowicz

## Information

I need to align tokens obtained with XLMRobertaTokenizer to original text. Normally, I use fast tokenizer API and token_to_chars method to get the mapping. However, when I use it with XLMRobertaTokenizer, there seems to be bug in the output (the returned indices of tokens tend to skip some tokens and sometimes do not correspond).

## To reproduce

```
from transformers import AutoTokenizer
input='walnut , 17.6.2007 22:20:59 , ip : *** . ***.108.25 , # 10305dobry den , nedavno me znicehonic zacal bolet nart prave nohy-spatne se s nim pohybuje , boli me pri chuzi .'
tokenizer = AutoTokenizer.from_pretrained('xlm-roberta-base', use_fast = True)
res = tokenizer(input_line)
print(res.tokens()[:5]
```

```
['<s>', '▁wal', 'nut', '▁', ',']
```

```
for i in range(15):
  cur_char_word_ind = res.char_to_token(i)
  print(input_line[i], cur_char_word_ind)
```

```
w 1
a 1
l 1
n 2
u 2
t 2
  None
, 3 
  None
1 5
7 5
. 6
6 6
. 6
2 7
```

The line ```, 3 ``` is wrong as ```,``` should be aligned to fourth token ```,```. The fourth token is not used and is skipped.

## Expected behavior

Possibly
```
w 1
a 1
l 1
n 2
u 2
t 2
  3
, 4 
  None
1 5
7 5
. 6
6 6
. 6
2 7
```

Indeed this seems a bit strange.

Pining @n1t0 and @Narsil here (actually this should probably rather be an issue in the https://github.com/huggingface/tokenizers repo)<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>@n1t0 Any update on this?<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.