GPT2Tokenizer <|endoftext|>
I am confused as to how the GPT2Tokenizer is intended to be used. It looks like `GPT2Tokenizer.encode` doesn't always take the byte pair encoding into account  -- is this intentional?

```
from pytorch_pretrained_bert import GPT2Tokenizer

tok = GPT2Tokenizer.from_pretrained("gpt2")

print(tok.encode("<|endoftext|>"))
print(tok.encoder["<|endoftext|>"])
```

Running the above gives:
```
[27, 91, 437, 1659, 5239, 91, 29]
50256
```
I would have (perhaps naively) expected that `tok.encode` gives `[50256]`. 
`"<|endoftext|>"` is a *special token* that is not intended to be feed through the tokenizer but added to the indices list after the tokenization process (see for example the way it is used in the short example [`run_gpt2.py`](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_gpt2.py#L98)).

It's common practice to distinguish the way we write *special tokens* for the constrains/behavior of the tokenizers as they (i) become unnecessary complex when we add exception rules for special tokens and (ii) are often designed externally (ex when you use SpaCy's tokenizer). The simplest way to do that is just to add the special tokens after the encoding process like it's done for GPT-2.<|||||>That makes sense! Is this in the documentation anywhere? I didn't come across it. Regardless, thanks for the clarification.<|||||>@thomwolf
I think after the new repo migration, you guys made '<|endoftext|>' as first class citizen of GPT2 tokenizer. So now we can just have base inputs with this token and it'd work fine with tokenizer and model, right?<|||||>Note that in the new version of Transformer, the behavior of the GPT2 tokenizer changed. 

```python
text = "<|endoftext|> machine learning using PyTorch and TensorFlow"
tokenizer_folder = "gpt2"
tokenizer = GPT2Tokenizer.from_pretrained(tokenizer_folder)
encoded_ids = tokenizer.encode(text)
print(encoded_ids)
print("################### when tokenizing the whole sentence ######################")
print(encoded_ids)
for id in encoded_ids:
    print(
        f"""id: {id:<7}, original_token: {"'" + tokenizer.decoder[id] + "'":<10}, ignore_special_token: '{tokenizer.decode(id)}'""")
# endfor
```

output: 
```text
################### when tokenizing the whole sentence ######################
[50256, 4572, 4673, 1262, 9485, 15884, 354, 290, 309, 22854, 37535]
id: 50256  , original_token: '<|endoftext|>', ignore_special_token: '<|endoftext|>'
id: 4572   , original_token: 'Ġmachine', ignore_special_token: ' machine'
id: 4673   , original_token: 'Ġlearning', ignore_special_token: ' learning'
id: 1262   , original_token: 'Ġusing'  , ignore_special_token: ' using'
id: 9485   , original_token: 'ĠPy'     , ignore_special_token: ' Py'
id: 15884  , original_token: 'Tor'     , ignore_special_token: 'Tor'
id: 354    , original_token: 'ch'      , ignore_special_token: 'ch'
id: 290    , original_token: 'Ġand'    , ignore_special_token: ' and'
id: 309    , original_token: 'ĠT'      , ignore_special_token: ' T'
id: 22854  , original_token: 'ensor'   , ignore_special_token: 'ensor'
id: 37535  , original_token: 'Flow'    , ignore_special_token: 'Flow'
```
