Thanks so much for taking the time to reply! Here are the results from my tests.



 thomwolf:

You probably have noted that the first word is a bit different because it’s lacking the first space but actually the model is trained like this and reach its best performances like this, with a special first word (see https://github.com/huggingface/transformers/issues/3788)


I guess that the results are better without a space mainly because that is the way GPT-2 was trained. Intuitively I would think it helpful for the model to know that “think” and " think" are directly related (we could even go further with capitalized versions, etc).
Something that surprised me is that, in the original training, if sentences are separated by new lines or just spaces, the tokenization will be very different (not just a new line token).



 thomwolf:

transformers thus provide an add_prefix_space argument to automatically add a space at the beginning if none is provided (more intuitive tokenization but slightly lower performances though).


I tested it and I’m also getting better results without adding extra space when fine-tuning on small tweets.
If we consider the 2 possible scenarios:

Scenario 1: training and prediction with <|endoftext|>token_without_space

Scenario 2: training and prediction with <|endoftext|> token_with_space


My intuition on the difference between these 2 scenarios is that the model will pull samples in probability from similar sequences.
There must be more samples of “Scenario 1” as I imagine most documents don’t start with a space, which as why we get better results.
We could also remove the <|endoftext|> token but in my tests we need to keep it as it probably also fulfills a “bos” function, letting the model know that we are starting a sample (learnt during fine-tuning).
I tried to create a new special token but results were much worse, probably because we need more data to learn its function and also because we lose in some way the pretrained knowledge where it was not present.
Let me know if you have any more insight. I’m really looking forward to large model training with a modified tokenizer, that would give as much info as possible at tokenization time to help the model.