Pre-tokenized inputs
The tokenizer also accept pre-tokenized inputs. This is particularly useful when you want to compute labels and extract
predictions in named entity recognition (NER) or
part-of-speech tagging (POS tagging).
Pre-tokenized does not mean your inputs are already tokenized (you wouldn't need to pass them through the tokenizer
if that was the case) but just split into words (which is often the first step in subword tokenization algorithms
like BPE).
If you want to use pre-tokenized inputs, just set is_split_into_words=True when passing your inputs to the
tokenizer. For instance, we have:
```py
encoded_input = tokenizer(["Hello", "I'm", "a", "single", "sentence"], is_split_into_words=True)
print(encoded_input)
{'input_ids': [101, 8667, 146, 112, 182, 170, 1423, 5650, 102],
 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0], 
