Has anyone used run_language_modelling.py to train a gpt 2 from scratch? 
I have read this post https://huggingface.co/blog/how-to-train and i would like to do the train from scratch, for a gpt2 type model. 
One question also, that i have, is the special tokens that he uses in that post for tokenizer will be the same and for a tokenizer that i will use for a gpt2 model?
Hi @nikkon3, the special tokens for `gpt2` are automatically set when you import `GPT2Tokenizer`.

The code below shows that `'<|endoftext|>'` is the special token used for BOS (beginning of sequence), EOS (end of sequence), and UNK (unknown - out of vocabulary).

```
from transformers import GPT2Tokenizer
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
tokenizer.special_tokens_map

#{'bos_token': '<|endoftext|>',
# 'eos_token': '<|endoftext|>',
# 'unk_token': '<|endoftext|>'}
```

Hope this helps you out!<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
