Extracting word representations from BPE-tokenization-based models (GPT-2, RoBERTa, etc.)
Hi! I am trying to extract representations from models (GPT-2 and RoBERTa) when the words are segmented into pieces. For this issue, let's assume I'm trying to extract the representation of `afoot` in the sentence: `the game is afoot !`

For this I first load my libraries and instantiate the RoBERTa tokenizer:

```py
import torch
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('roberta-base')
```
My inputs are in the form of `(sentence, idx)` where sentence is the context in which the desired word occurs in and idx which the index of the word (`afoot`) in the space-segmented form of the sentence `['the', 'game', 'is', 'afoot', '!']`: 

```py
sentence = ("The game is afoot !", 3)
```

The problem occurs when I compare the encoded token_ids for the word when it appears individually: 

```py
tokenizer.encode_plus('afoot', add_special_tokens = False)['input_ids']
#> [2001, 9210]
```

to the token_ids for the same word when it appears in a sentence:

```py
tokenizer.encode_plus('the game is afoot !', add_special_tokens = False)['input_ids']
#> [627, 177, 16, 10, 2917, 27785]
```

**Why do I even need the individually split token_ids?** 

Because I want to know which indices correspond to the desired word so that I can query the model when I pass the sentence to it as an input and extract the representations for the wordpieces/BPEs and then average them to represent the word's vector.

This is not a problem when I use BERT, where individual vs words as part of a sentence have the same segmentation.

Any ideas how I can solve this issue?

Environment:
```
python = 3.8.3
transformers = 3.1
torch = 1.6
```
SOLVED! Prefixing spaces before the word seems to do the trick for now. But I will wait to close the issue for people who have more elegant solutions to this problem.<|||||>> SOLVED! Prefixing spaces before the word seems to do the trick for now. But I will wait to close the issue for people who have more elegant solutions to this problem.

But how do you deal with the case that there are no spaces between two words?  For example
```
tokenizer.tokenize("I have a dog, which loves eating meat!")
#>  [['I', 'Ġhave', 'Ġa', 'Ġdog', ',', 'Ġwhich', 'Ġloves', 'Ġeat', 'ing', 'Ġmeat', '!']]
```

There is no space before ',' and '!', but both are words. How do you distinguish them from 'ing'? THX for your reply!