Fast Tokenizers instantiated via vocab/merge files do not respect skip_special_tokens=True
## Environment info
- `transformers` version: 4.3.2
- Platform: macOS-11.2.1-x86_64-i386-64bit
- Python version: 3.9.1
- PyTorch version (GPU?): 1.7.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

## Information

See title; this issue does not reproduce with slow tokenizers. Does not reproduce with serialized tokenizers.

Found while investigating https://github.com/minimaxir/aitextgen/issues/88

## To reproduce

Using [gpt2_merges.txt](https://github.com/minimaxir/aitextgen/blob/master/aitextgen/static/gpt2_merges.txt) and [gpt2_vocab.json](https://github.com/minimaxir/aitextgen/blob/master/aitextgen/static/gpt2_vocab.json) as linked:

```py
from transformers import AutoModelForCausalLM, GPT2Tokenizer, GPT2TokenizerFast

model = AutoModelForCausalLM.from_pretrained("distilgpt2")

outputs = model.generate(max_length=40)

# tensor([[50256,   383,   471,    13,    50,    13,  2732,   286,  4796,   468,
#           587, 10240,   262,  1918,   286,   257,  1966,  5349,  5797,   508,
#           373,  2823,   290,  2923,   416,   257, 23128,   287,   262,   471,
#            13,    50,    13, 13241,   319,  3583,    13,   198,   198,   198]])

tokenizer_fast = GPT2TokenizerFast(vocab_file="gpt2_vocab.json", merges_file="gpt2_merges.txt")
tokenizer_fast.decode(outputs[0], skip_special_tokens=True)

# '<|endoftext|> The U.S. Department of Justice has been investigating the death of a former FBI agent who was shot and killed by a gunman in the U.S. Capitol on Wednesday.\n\n\n'

tokenizer_slow = GPT2Tokenizer(vocab_file="gpt2_vocab.json", merges_file="gpt2_merges.txt")
tokenizer_slow.decode(outputs[0], skip_special_tokens=True)

# ' The U.S. Department of Justice has been investigating the death of a former FBI agent who was shot and killed by a gunman in the U.S. Capitol on Wednesday.\n\n\n'

```

Indeed, I can reproduce! Do you know what might be causing this @n1t0?<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>I think this is happening because when you load it from the vocab and merge files, it doesn't know `<|endoftext|>` is a special token. For the `skip_special_tokens` to work, I believe it would be necessary to add them to the tokenizer:
```python
tokenizer_fast.add_special_tokens({
    "additional_special_tokens": "<|endoftext|>"
})
```

The `tokenizer.json` file on the hub, available for `gpt2` does have this special token registered, that's why it works in this case.<|||||>That workaround is sufficient for my needs and appears to have done the trick. Thanks!