How to do the further pretraining ?
## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hello ! How could I futher Pre-train the BERT ( including the unsupervised masked language model
and next sentence prediction tasks ) **using my own corpus** ?   thank you very much !
We have no scripts for pre-training, but we do have scripts for fine-tuning (which seems to be what you want to do). Take a look at [run_lm_finetuning.py](https://github.com/huggingface/transformers/blob/master/examples/run_lm_finetuning.py) for more information. 

We don't have examples that do NSP however, as it was proven with RoBERTa to not be particularly useful for training. You'll have to code it yourself or find an implementation somewhere if you want to train on that loss.<|||||>@LysandreJik  I got it. thank you !<|||||>@JiangYanting If you want to do LM finetuning incl. NSP, you might wanna have a look at [FARM](https://github.com/deepset-ai/FARM). 
There's an example script [here](https://github.com/deepset-ai/FARM/blob/master/examples/lm_finetuning.py ).  

From our experience it depends a lot on the domain whether NSP makes sense. In some industry applications, we made good experience with also adding other auxiliary tasks in this phase of model training (e.g. an additional classification task for available tags of documents / sentences).

<|||||>@tholor  Wow, that's so cool ! I would have a try after I take a final exam^_^. thank you very much !<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
