Usage of Ġ in BPE tokenizer
Hello, 
I want to add new words to my BPE tokenizer. I know the symbol Ġ means the end of a new token and the majority of tokens in vocabs of pre-trained tokenizers start with Ġ. Assume I want to add the word **Salah** to my tokenizer. I tried to add both **Salah** token and **ĠSalah**:
`tokenizer.add_tokens(['Salah', 'ĠSalah'])` # they get 50265 and 50266 values respectively.
However, when I tokenize a sentence where **Salah** appears, the tokenizer will never return me the second number (neither when using `.tokenize` nor `.encode`), for instance:
`tokenizer.tokenize('I love Salah and salad')` returns `['I', 'Ġlove', 'Salah', 'Ġand', 'Ġsalad']`.
The question is: should I use the symbol `Ġ` when adding new tokens or the tokenizer does it itself? Or, probably, it must be specified manually?
Thanks in advance!
@maschasap 
AFAIK, you won't need to add `Ġ ` when adding a new token. 
And you can use  the `convert_tokens_to_string` method to convert these tokens to their respective strings.

tagging @mfuntowicz  for more info<|||||>@patil-suraj thanks for your response! But still why doesn't the tokenizer add `Ġ` symbol when returning the tokenized sentence in the example? Does it mean it still needs to be tuned or is it OK?<|||||>@patil-suraj btw using the method `convert_tokens_to_string` the whitespace between the words **love** and  **Salah** really disappears :(
`tokenizer.convert_tokens_to_string(tokenizer.tokenize('I love Salah and salad'))` outputs `'I loveSalah and salad`<|||||>@LysandreJik @mfuntowicz may I ask you for help?<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>Can someone help me understand the symbol Ġ ? I am running into the same thing , want to make sure I am not breaking things down the line<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>There is also a difference here between the fast and slow tokenisers:
```python
import transformers
tokeniser1 = transformers.RobertaTokenizer.from_pretrained('roberta-base')
tokeniser2 = transformers.RobertaTokenizerFast.from_pretrained('roberta-base')
tokeniser1.add_tokens(['Salah'])
tokeniser2.add_tokens(['Salah'])
print(tokeniser1.tokenize('I love Salah and salad'))
print(tokeniser2.tokenize('I love Salah and salad'))
```
Outputs:
```
['I', 'Ġlove', 'Salah', 'and', 'Ġsalad']
['I', 'Ġlove', 'Ġ', 'Salah', 'Ġand', 'Ġsalad']
```<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>same here<|||||>(in case someone comes across this issue, have a look at [this post in our forum](https://discuss.huggingface.co/t/bpe-tokenizers-and-spaces-before-words/475/2?u=joaogante))