Pre-training a smaller version of BERT on own data
## ❓ Questions & Help
Hello, 
<!-- A clear and concise description of the question. -->
I want to load the first 3-4 layers of BERT, pre-train those on my own data and then fine-tune the model on the target task. Is this possible?

I took a look at the [run_lm_finetuning.py](https://github.com/huggingface/transformers/blob/master/examples/run_lm_finetuning.py) and changed the lines 478-479:

`    config = config_class.from_pretrained(args.config_name if args.config_name else args.model_name_or_path,
                                          cache_dir=args.cache_dir if args.cache_dir else None)`
to
`    config = BertConfig(num_hidden_layers=3)
`
I'm not sure, if this change is sufficient and if the first or last three layers will be taken from the pre-trained BERT?

Is the _next sentence prediction_ pre-training task implemented somewhere ? 

Thanks in Advance 




Hi, you might be interested in freezing some layers rather than initializing a model with only 3 layers. Freezing some layers mean that they won't be affected by the backpropagation.

If that's what you're looking for, @BramVanroy's answer may be of help: https://github.com/huggingface/transformers/issues/1431

We do not have an NSP example in our examples.