Match the indexing output of `TokenClassificationPipeline` with labels (e.g. `ner_tags`)
### Feature request

As described below, currently there may well be cases where the token indexes given as an output of `TokenClassificationPipeline` can not be mapped to the original word indexes in the input dataset that originally had words already split.

It would come handy to be able to use the output indexes from the pipeline to retrieve e.g. the correct labels using the original dataset.

### Motivation

Currently, the preprocess function for `TokenClassificationPipeline` looks as follow:

https://github.com/huggingface/transformers/blob/215e0681e4c3f6ade6e219d022a5e640b42fcb76/src/transformers/pipelines/token_classification.py#L191-L205

An issue with datasets such as conll2003 where sentences are split by words is that to use the pipeline, we need to pass a single string:

```python
from transformers import TokenClassificationPipeline
from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForTokenClassification

model_name = "elastic/distilbert-base-uncased-finetuned-conll03-english"

model = AutoModelForTokenClassification.from_pretrained(model_name)
dataset = load_dataset("conll2003")
tokenizer = AutoTokenizer.from_pretrained(model_name)

pipe = TokenClassificationPipeline(model=model, tokenizer=tokenizer, aggregation_strategy="none", batch_size=2)

my_data = dataset["train"][4]
inputs = my_data["tokens"]  # this is a list of words

inputs = " ".join(inputs)  # we need to pass a single string to the pipeline as input

res = pipe(inputs)
print(res)
```

This is actually fine and everything works well (at the end of the day probabilities match), however, when using the `preprocess()` method as a standalone, there may be a slight issue:
```python
tokenized_inputs = pipe.preprocess(inputs)

print(inputs)
print(tokenized_inputs.word_ids(batch_index=0))
res = tokenizer.decode(tokenized_inputs["input_ids"][0])
print(res)
res = tokenizer.convert_ids_to_tokens(tokenized_inputs["input_ids"][0])
print(res)
"""
prints:

Germany 's representative to the European Union 's veterinary committee Werner Zwingmann said on Wednesday consumers should buy sheepmeat from countries other than Britain until the scientific advice was clearer .
[None, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 13, 13, 14, 15, 16, 17, 18, 19, 20, 20, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, None]
[CLS] germany's representative to the european union's veterinary committee werner zwingmann said on wednesday consumers should buy sheepmeat from countries other than britain until the scientific advice was clearer. [SEP]
['[CLS]', 'germany', "'", 's', 'representative', 'to', 'the', 'european', 'union', "'", 's', 'veterinary', 'committee', 'werner', 'z', '##wing', '##mann', 'said', 'on', 'wednesday', 'consumers', 'should', 'buy', 'sheep', '##me', '##at', 'from', 'countries', 'other', 'than', 'britain', 'until', 'the', 'scientific', 'advice', 'was', 'clearer', '.', '[SEP]']
"""
```

We see that `'` and `s` at the beginning of the sentence are mapped to different words (respectively 1 and 2). However, using

```python
# here we pass as input a list of words
tokenized_inputs = tokenizer(dataset["train"][4]["tokens"],
                             truncation=True, is_split_into_words=True)
```
we have that `'` and `s` are mapped to the same word (number 1).

This latter behavior is critical if we want to make use of the `ner_tags` with the pipeline (e.g. for evaluation). Indeed, by assigning to different words tokens that were originally part of the same word (before joining with `" ".join()`), we break the indexing given in the pipeline output `index`, in the sense that those token indexes can no longer be mapped to the original `ner_tags` (which uses word indexes).

This mapping normally could be done using the `tokenized_inputs.word_ids`.

**Motivation:** Use pipelines for the evaluation in Optimum.

**Alternative approach:**

We could use `tokenizer._tokenizer.pre_tokenizer = pre_tokenizers.Sequence([WhitespaceSplit()])` to explicity avoid splitting on punctuation, however I guess it is bad practice to overwrite a pre-tokenizer (in my case [BertPreTokenizer](https://github.com/huggingface/tokenizers/blob/28cd3dce2a75d106572392194ff2564574c33235/tokenizers/src/pre_tokenizers/bert.rs#L13))?

### Your contribution

Not sure how to go about this. We could support `is_split_into_words` argument for `preprocess()`, but then this method would accept list of words, which is not the case right now (only full sentence as a single string).
Maybe of interest to @Narsil <|||||>Hi @fxmarty 

> however I guess it is bad practice to overwrite a pre-tokenizer (in my case [BertPreTokenizer](https://github.com/huggingface/tokenizers/blob/28cd3dce2a75d106572392194ff2564574c33235/tokenizers/src/pre_tokenizers/bert.rs#L13))?

Yes, this would set up for shooting yourself in the foot later :)

> Not sure how to go about this. We could support is_split_into_words argument for preprocess(), but then this method would accept list of words, which is not the case right now (only full sentence as a single string).

I am not sure this would even work, since the tokenizer itself would end up splitting things anyway. (It might but I am not sure, and using those usually come with their own caveat, so I usually try to avoid them)

> inputs = " ".join(inputs)  # we need to pass a single string to the pipeline as input
Since you already have your tokens, can't you use directly them for token number (I refrain to use `word_ids` on purpose ;) ).

And then maybe do something like using `offset_mapping` to map the output of the tokenizer and the pretokenized thing (handling conflicts which might occur), usually they are the best way to map what's incoming to the `input_ids`.


```python
def get_conll_word_ids(input_ids, offset_mappings, conll_tokens):
    token_index = 0
    sentence_index = 0
    word_ids = []
    for i, (input_ids, (start, stop) ) in enumerate(zip(input_ids, offset_mappings)):
        while start > sentence_index:
           # This is a new conll token
           sentence_index += conll_tokens[token_index] + 1 # +1 is The inlined space
           token_index += 1
       word_ids.append(token_index)
```

This doesn't handle conflict which might exist if `start` and `stop` actually overlap 2 or more different conll tokens.
This code is a bit specific for conll kind of task, however I don't think there's a nice workaround anyway since what the `tokenizer` sees may be entirely different from what the conll tokens look like and using `pretokenized` sentences might actually induce biases into what the model sees leading to poorer than expected performance (depends on how exactly the training loop was done and if that bias was also put into that model, but I am guessing usually it's not done that way)<|||||>Thanks a lot for your help! In the meantime, I implemented my own trick to avoid modifying the pre-tokenizer, basically tokenizing word by word first to build a good `token_to_word_id`:

```python
inputs = " ".join(data["tokens])  # where `data` is a sample from the dataset
res = pipeline(inputs)

# BatchEncoding.word_ids may be wrong as we joined words with " ", so let's populate it ourselves
token_to_word_id = []
for j, word in enumerate(data["tokens"]):
    preprocessed_inputs = pipeline.preprocess(word)
    n_tokens = len([k for k in preprocessed_inputs.word_ids(0) if k != None])  # exclude None
    token_to_word_id.extend([j] * n_tokens)

# the pipeline may give as output labeled tokens that are part of the same word, keep track
# of the indexing to match the true labels on words
index_tokens_word_start = []

for j, word_index in enumerate(token_to_word_id):
    if j == 0:
        index_tokens_word_start.append(j)
    elif word_index != token_to_word_id[j - 1]:
        index_tokens_word_start.append(j)

# keep only predictions that correspond to the beginning of a word
preds = [res[index]["entity"] for index in index_tokens_word_start]
```
<|||||>Hi @fxmarty,

Great that you found a workaround, which probably works OK for you combination of model+dataset+metric. I would like to point however that it's probably not extremely generic.

Take the following with a  big grain of salt since the following is a personal opinion:

> keep only predictions that correspond to the beginning of a word

That works for things like conll maybe, but what if the model actually split a conll word into 2 tokens and give them 2 different token classes, is it really OK ? I would argue that it's not, even if I know some researchers do that to boost results. It's an issue since why would the first token yield more information than the second ?  What if the model is a byte level one like ByT5 and every letter has an entity, is really the first letter the best choice ? Other aggregation strategies exist in the pipeline, which might yield better results for some models, but I think all are slightly lying. It makes sense for the pipeline since, as a user in an app, or webapp, displaying part of words as entities is super odd in space spitted languages, however if I am evaluating two different models, I think that evaluating `[Correct, Incorrect]` the same as `[incorrect, correct]` (if the word is split in two tokens) seems important IMO. If some byte level model comes along that predicts `[Incorrect, correct, correct, correct]` my take is that is should get a better score than the previous two.

The pipeline does provide `aggregation_strategy="first"/"max"/"avg"`  which does come with caveat (and for instance will never work on non space separated languages like Chinese where's there's no real word boundary of words.

This is even emphasized for multi word tokens in NER where "New York" and having entities be "B-LOC", "I-LOC", "I-LOC" is the only way to get a single "New York" entity across words.

Using `offset_mappings` is IMO the only viable way to get a robust alignment of tokens for two different tokenizations. 
Going that route, you can directly use the various `aggregation_strategy` (even if they work only on space separated languages). 
Ofc many conflicts can (and will) occur, but at least you can define 1 single consistent strategy to deal with those.

Another suggestion would be to start right of the bat on Chinese or Thai for instance, which will trigger a lot more questions than space separated languages, and usually answers for these languages tend to be correct for simpler languages like English (and more correct that what the naive thing for English would be). And the solutions tend to be simpler too.

Looking at what the issue you linked, maybe enabling users to write custom script (akin to GH actions maybe ?) might be the best solution. (We could write a sane default Evaluate that aims to be generic, but users could always entirely define their own)<|||||>Hello Narsil, thanks a lot for your detailed feedback, this is really appreciated.

So I think what you suggest is that the flag `label_all_tokens` should be `True` to get a fair evaluation in the token classification example. https://github.com/huggingface/transformers/blob/b9bb417324c0d9013c505dc39c016ab9ca0e23c8/examples/pytorch/token-classification/run_ner.py#L426-L445

For now, I was merely trying to reproduce the PyToch example scripts using pipeline for evaluation so I just went for the default `label_all_tokens=False`, where indeed only the first token of a word is used for evaluation.

> It makes sense for the pipeline since, as a user in an app, or webapp, displaying part of words as entities is super odd in space spitted languages, however if I am evaluating two different models

Not sure about what you meant there, but actually the pipeline may return different labels for the same word! See e.g. https://huggingface.co/docs/transformers/task_summary#named-entity-recognition , and I recall reading at some point (in the course or doc) that it was probably better not to abstract away too much, at the expense of giving several labels for the same word.

Your thoughts about different predictions for tokens of the same word got me thinking if this was not as well dependent on how the model was trained. I am not familiar at all with token classification (so my understanding may be broken), but naively *"Why evaluate on first token if the model was trained on all"*, and conversely *"Why evaluate on all tokens if the model was trained on only the first"*.

> Using offset_mappings is IMO the only viable way to get a robust alignment of tokens for two different tokenizations.
Going that route, you can directly use the various aggregation_strategy (even if they work only on space separated languages).
> Ofc many conflicts can (and will) occur, but at least you can define 1 single consistent strategy to deal with those.
>
> Another suggestion would be to start right of the bat on Chinese or Thai for instance, which will trigger a lot more questions than space separated languages, and usually answers for these languages tend to be correct for simpler languages like English (and more correct that what the naive thing for English would be). And the solutions tend to be simpler too.

Anyways, I will have a deeper look at your two suggestions. One thing I was wondering today is how to deal with datasets as https://huggingface.co/datasets/msra_ner , if it makes sense to join by space at all, if it could hurt performances (see e.g. https://huggingface.co/abhishek/autonlp-japanese-sentiment-59363/blob/main/vocab.txt#L23394 where several characters can be a token, if my understanding is correct).

Out of curiosity, are there other cases where you think my approach (tokenize each word once first) would break?

Again, thank you for your feedback, it is really helpful.<|||||>> Out of curiosity, are there other cases where you think my approach (tokenize each word once first) would break?

I think it's pretty standard and definitely has merits which I didn't necessarily restated clearly enough.And it IS used as you suggested, so reusing it if that's how the model was trained is the best option. But you don't necessarily know how a model was trained i think. Also I think it's not necessarily super general and the odd case would pop up fast if you wanted to try to evaluate some random model which has a very different tokenization for instance.

Again my personal take would be to take directly the most general case first and on possibly a non separated language.

In `tokenizers` at least going the general route was often the easiest in the long run.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.