question about 'add_prefix_space' of encode method
## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

Hi, I have a question about the 'add_prefix_space' parameter of encode method. I'll use small gpt2 for example.

> tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
> print(tokenizer.encode("my",add_prefix_space=True)) #616
> print(tokenizer.encode("my",add_prefix_space=False)) #1820

I think 616 is the index for the word 'my'(' my') since there's a space before letter 'm'. 
And 1820 is the index for '-me' which is part of a word. Am I right? 

Thank you!
Actually, both 616 and 1820 are indices for `my`. The difference lies in the prefix space that was added: in the case that it was not added (1820), it is identified as being the beginning of the sentence or part of a word.

In the case that it was added (616), it is identified as being the beginning of a word in a sentence.

You can check the behavior by calling `tokenize` instead of `encode`, for example:

```py
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')

print(tokenizer.tokenize("my son is jeremy",add_prefix_space=True))
# ['Ġmy', 'Ġson', 'Ġis', 'Ġj', 'ere', 'my']

print(tokenizer.tokenize("my son is my son",add_prefix_space=False))
# ['my', 'Ġson', 'Ġis', 'Ġmy', 'Ġson']
```<|||||>Thank you! <|||||>still don't understand what is the difference in 
```
tokenizer = tokenizers.ByteLevelBPETokenizer(
    vocab_file=PATH+'vocab-roberta-base.json', 
    merges_file=PATH+'merges-roberta-base.txt',
    lowercase=True,
    add_prefix_space=True
)
```<|||||>The argument you mention is for the initialization of a tokenizer from the `huggingface/tokenizers` library, you would have better luck opening an issue there.