Recommended preprocessing steps for english sentences in GPT2
# ❓ Questions & Help

To run inferences in english sentences... I'm not really sure what preprocessing steps I need to do before sending the tokenized text to gpt-2 in order to get the predictions. Any advice?

How to handle non-english words that can be found in an english sentences? Extra punctuation, space, new lines, user's mentions, hashtags, urls, alternative apostrophes, etc...?

## Details
<!-- Description of your issue -->

**A link to original question on Stack Overflow**:  https://stackoverflow.com/questions/60807799/gpt2-huggingfaces-transformer-preprocessing
Just pre-process them as you would for any other NLP task. This may include: normalisation of punctuation, dealing with HTML/XML, white-space normalisation, language verification. Something that people used to do in the age of word2vec is creating new entities for specific items. For instance, any number could be replaced with all 1s (1234 becomes 1111) to have relatively small vocabulary that still takes the number of characters into account. Same with users (e.g @user@) and mentions and urls (@URL@), and so on. This might still be worth the effort, but not necessarily. In such cases you may want to ensure that these tokens are not split in the tokenizer.

I hope that you understand that this is a very general question not related to this repository, so I am closing this.<|||||>@BramVanroy I think I forgot to mention that I'm using the gpt-2 training models from this repo. I'm not re-training gpt-2. I think I shouldn't create new entities for specific terms if that process has not happened during training step. Am I right? <|||||>If you are not planning to pretraining the model (and generating a new vocab) then, indeed, you should not try to add new tokens. So in your case you would just need to do some basic normalisation of punctuation, HTML, etc.<|||||>@BramVanroy Thanks for the answer. I'm directly using gpt-2 pretrained models from https://huggingface.co/models . Specifically the ones that are created by Huggingface. I can't find the preprocessing code that was used when those models were trained... so I can replicate the same at inference time. I'm wondering if I should just assume things or it'd be safer to see how it was trained so I can make sure to prepare the sentences in a similar way.<|||||>Has there been any preprocessing during training phase @BramVanroy ? <|||||>I don't know. Note that HuggingFace didn't train GPT2. They ported the weights to their own architecture. You can try to get into contact with the people that created it, OpenAI. https://github.com/openai/gpt-2<|||||>@Damiox have you found the original preprocessing code?<|||||>@don-prog no, I have not 😢 - I am doing some subset of the preprocessing heuristics that @BramVanroy detailed before when serving the model. But I still think it'd be really good to have a consistency between both preprocessing mechanisms: training (whatever it was) vs inference. I just haven't had the time to identify that original preprocessing code from OpenAI