integer representation ambuiguty in tokenizer
## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

I use GPT-2 transformers model. Tokenizer.encode(' man') = 805, tokenizer.encode('man') = 805. But when in the sentence(e.g. Tokenizer.encode(' the man is a teacher') = [1169, 582, 318, 257, 4701], here the integer representing 'man' is 582. I think the problem is BPE used in transformer, where 850 is the integer representing the subtoken '-man' but not the word 'man'.
I wonder how I can set the tokenizer so I can get Tokenizer.encode(' man') = 582?
Hello! You should specify `add_prefix_space=True` in your encode method to obtain that behavior.<|||||>Thank you! That works!