I would like to use transformers/hugging face library to further pretrain BERT. I found the masked LM/ pretrain model, and a usage example 95, but not a training example.
In the original BERT repo I have this explanation 95, which is great, but I would like to use Pytorch.
I’m not looking to finetune the model, just pretrain it further on the IMDB dataset, starting with an already trained model.
Also, I assume that extracting BERT weights without the masked LM head and loading it into a regular BERT model should not be an issue.