Hi @rgwatwormhill,
Thank you for the response.  Even though you preface your post by claiming you are not an expert, I appreciate your post nonetheless.
I asked a question to the forum earlier (it went unanswered) related to replacing special characters or bizarre strings with understandable strings. For instance, I figured a hyperlink that goes https://www 2… would not make much sense to Bert, but the word hyperlink might. This is similar with what you were saying: maybe replacing the numbers with words might make a positive impact.
I think the only way to find this out will be to try it both ways and see what happens.  I’m sure there are a few people out there that have conclusive answers to this, but I’m not sure they will respond to this thread.
I recently stumbled on this video that came out almost a year ago. The video goes over wordpiece embeddings, but it also goes over what types ok tokens appear in the vocabulary. See here: https://youtu.be/zJW57aCBCTk?t=906 14
It confirms that there are characters from non-English languages, symbols, numbers, and other special characters in the base version of bert.
I was thinking in terms of training my own model using MLM, how much should I remove? I’ll probably just have to try and see.