Is there any pre-training example code?
I want to pre-train BERT with my own data.
[#124](https://github.com/huggingface/pytorch-pretrained-BERT/pull/124), [#170](https://github.com/huggingface/pytorch-pretrained-BERT/issues/170) say the model can pre-training.
But I can't find pre-training example code.
Is there any pre-training example code?
If there is no example code, I wonder [modeling.py](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/pytorch_pretrained_bert/modeling.py)'s BertForPreTraining is enought to pre-tranining.
Hi @tmchojo there are now several detailed examples [here](https://github.com/huggingface/pytorch-pretrained-BERT/tree/master/examples/lm_finetuning) thanks to @Rocketknight1 and #392 <|||||>Thanks.
But it still needs pre-trained model.
My data is another domain (not English or Chinese), so I can't use pre-trained model.
Do I have to use [Google's](https://github.com/google-research/bert) to pre-train?<|||||>I'm also interested in pre-training. Did you make any progress on this, @tmchojo?<|||||>There is an example provided [here](https://huggingface.co/blog/how-to-train) with colab (this is for pre-training from scratch), however, I'm still trying to figure the continual pre-training and not pretraining from scratch (using a pre-trained model as initial checkpoint). <|||||>Hope https://github.com/huggingface/transformers/tree/master/examples/pytorch helps! The language modelling notebook trains  a gpt model. This notebook also gives an idea on labelling the dataset for the LM task.
