Bug in TokenClassificationPipeline
## Environment info

- `transformers` version: 4.5.1
- Platform: Linux-5.4.0-42-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.10
- PyTorch version (GPU?): 1.8.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
@Narsil , @LysandreJik 

## Information

The problem is in TokenClassificationPipeline, in and [this](https://github.com/huggingface/transformers/blob/f4a0d6ff867e8a82a33d7a653e7d45372a463271/src/transformers/pipelines/token_classification.py#L269) and [that line](https://github.com/huggingface/transformers/blob/f4a0d6ff867e8a82a33d7a653e7d45372a463271/src/transformers/pipelines/token_classification.py#L273) .
Here the aim is to determine if the original word for that token is tokenized into multiple subwords or just a single one. The problem is some tokenizers (such as Roberta or GPT-2) tokenize the whitespace together with the subsequent word which causes a mismatch between the original word and the reconstructed word. Since we reconstruct from the tokenized input ids, a single-word token also includes a whitespace (unles it is not the first word in the sequence). 

## To reproduce
Please, consider the following:

```python
from transformers import AutoTokenizer
word_ref = "Car"
tokenizer = AutoTokenizer.from_pretrained("roberta-base")
word = tokenizer.tokenize(" " + word_ref)[0]
print(word)
>>> ĠCar
is_subword = len(word_ref) != len(word)
print(is_subword)
>>> True
```
The problem I simulated occurs in my custom pipeline that inherits from TokenClassificationPipeline when I use Roberta tokenizer. I checked the tests for that pipeline and observed that a small Bert tokenizer is used. This can explain why this bug could not be catched as the Bert model tokenizes the spaces differently. If I recall correctly, it splits the words on the whitespaces, then tokenizes the words. In any case, the following result shows why Bert tokenizer does not suffer from the mentioned problem:
```python
from transformers import AutoTokenizer
word_ref = "Car"
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
word = tokenizer.tokenize(" " + word_ref)[0]
print(word)
>>> Car
is_subword = len(word_ref) != len(word)
print(is_subword)
>>> False
```

Finally, this problem might be affecting other pipelines (or inference scripts etc.) that depends on the reconstructed tokens as well.

That is true. `len(word_ref) != len(word)` is a heuristic that will work on tokenizers that use BPE `continuing_subword_prefix` concept.
 
The reality is that there is no consistent notion of a "word" within arbitrary tokenizers.
The `continuing_subword_prefix` in BPE that *can* be used makes the concept explicit but it's not in the case of GPT2 (and roberta-large) as they are supposed to be ByteLevel. (it is set to '').

Because of that, there cannot be any consistent manner to check for "is_subword"  for these tokenizers.

Let's take an example "Hello thereHello" with `roberta-large`.

-> [    0 31414    89 31414     2]

We have twice the same token (31414), one is not a subword, the second one is. So there can't be a perfect output in any case.
Token 89 is really "  there" the space isn't treated that differently any other characters.

Is that clearer on why it fails in this use-case ?

That being said, if we can figure out a heuristic that works for both, it would be better indeed.
<|||||>When I had a similar problem, I resolved it by checking the character before that word in the original string. In this case, if there is a space, we can include it to the word, tokenize it and join them into a single string. The `word_ref` would be changed like the follows:

```python
if start_ind > 0 and sentence[start_ind-1] == " ":
    decoded_word_ref = "".join(self.tokenizer.tokenize(sentence[start_ind-1: end_ind]))
else:
    decoded_word_ref = sentence[start_ind:end_ind]
```

(I am replacing the identifier `word_ref` with `decoded_word_ref` to emphasize that it is reconstructed from the token ids and may not correspond to a valid substring in the original text)
Therefore, the related code segment would be updated as follows:

```python
                    if start_ind > 0 and sentence[start_ind-1] == " ":
                        decoded_word_ref = "".join(self.tokenizer.tokenize(sentence[start_ind-1: end_ind]))
                    else:
                        decoded_word_ref = sentence[start_ind:end_ind]
                    word = self.tokenizer.convert_ids_to_tokens([int(input_ids[idx])])[0]
                    is_subword = len(decoded_word_ref) != len(word)

                    if int(input_ids[idx]) == self.tokenizer.unk_token_id:
                        word = decoded_word_ref
                        is_subword = False
``` 

Notice that, even if there were multiple whitespaces before the word, they should not cause an issue since each space would be tokenized as a separate token except the last one.

Alternatively, we might use the decoded_word_ref only for determining the value of `is_subword`. After that, we can use the `word_ref` as before.
<|||||>That wouldn't work because some byte-level tokenizers will use space as a postfix, not prefix for "word-separation". 
This is where we would like to avoid many if conditions for every possible tokenizer.
<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.