[TBD] discrepancy regarding the `tokenize` method behavior - should the unknown token be included or not
It seems that not all tokenizers have the same behavior regarding the `tokenize` method, in particular some tokenizers will show the token unknow if the token does not belong to the vocabulary and other tokenizers will show the snippet corresponding to this token unknow in the initial text.

I've created a [google colab](https://colab.research.google.com/drive/1lptT0LXsO1B9QCrOL439SHxdeFJMFe8V?usp=sharing) showing the output of the `tokenizer.tokenize(text)` method and `tokenizer.convert_ids_to_tokens(tokenizer.encode(text, add_special_tokens=False))` on several tokenizers.

I have the impression that this difference is related to the tokenization algorithm used (and therefore probably to the behavior in the `tokenizers` library).

This raises several questions:
- for what reasons is this `tokenize` method used by users.
- is it upsetting to have multiple behaviors? 
- what is the right behavior?
- if we choose to standardize this behavior, will it cause BC problems?
Hi @SaulLu, thank you for opening an issue! IMO the `tokenize` methods should behave somewhat consistently across tokenizers - but from what I understood this may be impossible given the different tokenization methods.

My opinion on the matter is:
- We should try to have all `tokenize` methods conform
- If some cannot, then clearly document them
- If we do some changes, these will have to remain backward-compatible until v5, but we can have a v5 major version that *slightly* changes the behavior if we document each of them clearly. <|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.