Multiple token IDs for same token
## ❓ Questions & Help

I am using GPT2Tokenizer. I observed that some tokens are duplicated in the vocabulary with a space appended to them in the beginning. For example, there exist two separate tokens - `can` and `<space>can`. Both of them are mapped to different token IDs - `5171` and `460` respectively.

Although, both - 
```
can_token_id = tokenizer.encode(' can', add_special_tokens=False)
```
and
```
can_token_id = tokenizer.encode('can', add_special_tokens=False)
```
return can_token_id as `[5171]`

Is this an intended property of the tokenizer? Is there a mapping available between what token IDs are just a space appended version of other tokenIDs?
Okay, I found that the symbol is not `<space>` but this `Ġ`
So there exist two words in the vocab - `can` and `Ġcan`
Why is this so?<|||||>Hi! This is indeed an intended property of the tokenizer. The GPT-2 tokenizer is a byte-level BPE that has a sufficient vocabulary size to make the distinction between tokens that are at the beginning of a sentence (not prepended by a space), and those that are in the middle of a sentence (prepended by a space). 

The tokens `the`  and ` the` are therefore encoded differently, however the tokenizer strips the spaces from the sequences it receives as input. You can suppress that behavior by setting the `add_prefix_space` flag to `True`:

```py
tokenizer.encode("the")
# [1169]

tokenizer.encode("the", add_prefix_space=True)
# [262]
```

Concerning your question regarding why the vocabulary displays `can` and `Ġcan`, you can actually see the `Ġcan` as being ` can` (notice the space). The GPT-2 tokenizer converts all spaces/control characters to other tokens. These spaces and control characters could have some unwanted behavior when using a BPE tokenizer (for example if it splits on whitespace).

The space token is switched to `Ġ`. You can see it being done using the `bytes_to_unicode` method in the [tokenization_gpt2.py](https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_gpt2.py#L63-L85) file.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
