I know that BERT has tokens for numbers, punctuation, and special characters (e.g. #@!%). If I’m training a language model, should I

Keep numbers, punctuation, and special characters
Remove only the aforementioned characters, leaving the rest of the sentence untouched
Remove the whole sentence if it contains any of those.