Tokenizer's normalization preprocessor cause misalignment in return_offsets_mapping for tokenizer classification task
## Environment info
- `transformers` version: 4.6.0
- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.10
- PyTorch version (GPU?): 1.8.1+cu102 (False)
- Tensorflow version (GPU?): 2.5.0 (False)
- Using GPU in script?: not relevant
- Using distributed or parallel set-up in script?: not relevant

### Who can help
Models:
- albert, bert, xlm: @LysandreJik


Library:
- tokenizers: @LysandreJik


 -->

## Information

Model I am using "xlm-roberta"

The problem arises when using:
I am using my own dataset building script in the provided example, but the script should be equivalent to the changes made by this [update](https://github.com/huggingface/datasets/pull/2466)
`get_dataset `is just a simple wrapping for `load_dataset`
and the `tokenizer` is just `XLMRobertaTokenizerFast.from_pretrained("xlm-roberta-large")`

The tasks I am working on is:
Xtreme udpos dataset (or potentially any other multilingual token classification task)

## To reproduce

[This colab notebook](https://colab.research.google.com/drive/151gKyo0YIwnlznrOHst23oYH_a3mAe3Z?usp=sharing) implements a token classification input pipeline extending the logic from [this hugging example](https://huggingface.co/transformers/custom_datasets.html#tok-ner).

The pipeline works fine with most instance in different languages, but unfortunately, [the Japanese Kana ligature (a form of abbreviation? I don't know Japanese well)](https://en.wikipedia.org/wiki/Kana_ligature) break the alignment of `return_offsets_mapping`:
![image](https://user-images.githubusercontent.com/50871412/122904371-db192700-d382-11eb-8917-1775db76db69.png)

Without the try catch block, it riase `ValueError: NumPy boolean array indexing assignment cannot assign 88 input values to the 87 output values where the mask is true`, example shown here [(another colab notebook)](https://colab.research.google.com/drive/1ZGj-4LzhnjrDv3PC5nlmi-BVj9vgvFTp?usp=sharing)
```
/content/MLM-disentangle/experiment_datasets/xtreme_ds.py in __getitem__(self, id_absolute)
    605         labels[
    606             (arr_offset[:, 0] == 0) & (arr_offset[:, 1] != 0) & (ids[:] != 6)
--> 607         ] = self.dataset[lan][id]["pos_tags"]
    608         return {
    609             "tokens": torch.from_numpy(ids).long(),

ValueError: NumPy boolean array indexing assignment cannot assign 88 input values to the 87 output values where the mask is true
```

It is clear that the normalizer is the process that break the alignment, as it is observed that `tokenizer._tokenizer.normalizer.normalize_str('ヿ')` return 'コト'. And both the tokens, 'コ' and  'ト', have True value when evaluating the `(arr_offset[:, 0] == 0) & (arr_offset[:, 1] != 0) & (ids[:] != 6) ` logic , which breaks the alignment of `return_offsets_mapping`.

## Expected behavior

One workaround is to include `tokenizer._tokenizer.normalizer.normalize_str` before the tokenizer preprocessing pipeline, which is also provided in the [first colab notebook](https://colab.research.google.com/drive/151gKyo0YIwnlznrOHst23oYH_a3mAe3Z?usp=sharing) with the name `udposTestDatasetWorkaround`.

I guess similar logics should be included inside the tokenizer and the offsets_mapping generation process such that user don't need to include them in their code. But I don't understand the code of tokenizer well that I think I am not able to do this.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.