Can we have a way for a tokenizer to transform word level or character level annotations?
# 🚀 Feature request

If I have a string with some annotations and I want to tokenize it I'd like the tokenizer to be able to transform the annotations as well. For example suppose I have `s = "therein lies the problem."` and I'm interested in the substring `"the problem"`. So I have a string `s` and I know that the substring I'm interested in is at index 13:24. But then I tokenize `s` so that I can put it into a huggingface model and get out `['there', '##in', 'lies', 'the', 'problem', '.']` and it doesn't match up with my annotation anymore. Could we add an annotation as an additional argument to the tokenizer.tokenize function so that I could get something like the following:

```
tokenizer.tokenize("therein lies the problem.", selection=(13,24))
> ['there', '##in', 'lies', 'the', 'problem', '.'], [0, 0, 0, 1, 1, 0]
```

Substrings of interest are almost always going to line up with token boundaries so it doesn't matter too much what happens to a token that is partially in and partially outside of the selected region. 

Is there a way of doing something like this now?

I just found out about the `return_offsets_mapping` functionality in the `PreTrainedTokenizerFast` tokenizers. I think I can use that functionality to solve my problem. <|||||>@brian8128 could you share a code example of how you do that?<|||||>Hey Avijit - I didn't test this because it's sort of copied from various places in my codebase but here's the general idea. There may be multiple different char level labels within one token so here I take the most common one using the scipy.stats.mode function. 

```
from scipy.stats import mode
from transformers import BertTokenizerFast

texts = ... # list of strings
char_level_labels = ... # list of 1-d numpy arrays corresponding in length to texts

tokenizer = BertTokenizerFast.from_pretrained('bert-base-uncased')
batch = tokenizer.batch_encode_plus(texts, return_offsets_mapping=True)

offset_mappings = batch['offset_mapping']

token_level_labels = []
for offset_mapping, char_level_label in zip(offset_mappings, char_level_labels):
    token_level_label = []
    for so, eo in offset_mapping:
        # Huggingface adds a start and end token that don't correspond to any
        # chars, we'll label these tokens with -1
        label = mode(char_level_label[so:eo]).mode[0] if eo - so > 0 else -1
        token_level_label.append(label)

    token_level_labels.append(np.array(token_level_labels))
```