Code bug in tokenization_utils.py?
In the function split_on_tokens in tokenization_utils.py, it contains the following logic:

if not text.strip():
    return []
if not tok_list:
    return self._tokenize(text)

So if the text contains just white space, '[]' will be returned. However, if the text contains white space and visible characters, the white space will be used in tokenization. For example, for the following code:

from transformers import GPT2Tokenizer
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
print(tokenizer("\nNorth")['input_ids'])  # output [198, 14157], since 198 <-> \n and 14157 <-> North
print(tokenizer("\n")['input_ids'])            # output [], even if 198 <-> \n

Is this behavior that we expected?
Pinging @mfuntowicz, @n1t0 for their opinions<|||||>I think we should delegate to the underlying tokenization algorithm and never strip like we do here. This what is done in `tokenizers` and thus one of the source of discrepancies between fast and slow tokenizers.

For most tokenizers this is not a problem since these white space get removed later anyway, but for some others (like gpt2) it is.

Note: it is impossible to build tokenizers relying on the formatting with such rules. For example it would be impossible to tokenize some Python.

(Cc @thomwolf)<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
