Help with finetune BERT pretraining 
Hi
could you please assist me how I can pretrain the BERT model, so not like SNLI/MNLI when we finetune a pretrained model, but doing pretraining objective.
thanks a lot
and Merry Christmas and happy new year in advance to the team
I suggest you follow the [run_lm_finetuning.py](https://github.com/huggingface/transformers/blob/master/examples/run_lm_finetuning.py) script. Instead of downloading a pretrained model, simply start with a fresh one. Here's an example:

```
from transformers import BertModel, BertConfig, BertTokenizer
model = BertModel(BertConfig())
tokenizer = BertTokenizer.from_pretrained('bert-base-cased')
```

I personally don't have a reason to write my own tokenizer, but if you do feel free to do that as well. All you need to do is generate a vocab.txt file from your corpus.<|||||>You can now leave `--model_name_or_path` to None in `run_language_modeling.py` to train a model from scratch.

See also https://huggingface.co/blog/how-to-train