hf4nlp:

Thank you @neuralpat , but is it really that simple? I mean does the MLM head implement the BERT pre-training algorithm, masks the 15% of the input tokens etc., and continue the pre-training of the model?


Yes, I believe so. I could be wrong, I’m kinda new to this myself, but  I don’t see anything indicating otherwise. It being “too easy” is generally a feeling I get when using huggingface. It feels like cheating 
You can test all the things you’re thinking about. Just MLM, MLM and NSP, just fine-tuning…What works best is usually only found out empirically.