I believe these two tutorials should be helpful:


Main NLP tasks - Hugging Face Course. This one trains a Causal LM from scratch, whereas you seem more interested in training a Masked LM (e.g. BERT) from scratch. That being said, many of the same principles apply.

How to train a new language model from scratch using Transformers and Tokenizers 1.

Hope those are helpful!
Cheers
Heiko