LM fine-tuning without NSP
Hello,

I'm thinking about fine-tuning a BERT model using only the Masked LM pre-training objective, and I'd appreciate a bit of guidance. The most straightforward way is probably to modify the simple_lm_finetuning.py script to only do LM fine-tuning. Besides importing BertForMaskedLM instead of BertForPretraining (which has both objectives), what changes should I make, and what potential problems should I consider?

Also, would it make sense to do MLM fine-tuning on a relatively small domain-specific corpus consisting of just the training examples from the datasets? In other words, does it make sense to do LM pretraining on a corpus that includes training examples taken from the datasets for which you want to make downstream predictions? I'm assuming including the dev/test examples in the corpus is out of the question since it'll likely overfit, but what about just the training examples? I'd like hear to people's thoughts about it.

Lastly, was the run_lm_finetuning.py script replaced by the contents in the examples/lm_finetuning directory? If so, there are still 2 references to that script in the README that should be edited. 

@Rocketknight1 
To your first question, the inputs will be almost identical, but the token_type_ids argument will be unused, as this is the vector that indicates the split between the two 'sentences' for the NextSentence objective. I'm not familiar with that part of the code - you might be able to just pass `None` for that argument, or maybe you'll need to pass a vector of zeros that has the right shape (i.e. the same length as the sequence). Also, I believe pre-training with just the masked LM objective works okay - it's by far the most important of the two objectives. The BERT paper didn't show any ablation studies, so I don't know what the effect of removing it will be, but I wouldn't worry too much for a fine-tuning task.

Secondly, I wouldn't expect a large benefit from just fine-tuning the LM on a small labelled training corpus. The main benefit arises when you have a large corpus to fine-tune the LM on, but only a small fraction of it is labelled. 

Also, I would guess that fine-tuning the LM on the dev/test examples will probably result in test performance that is optimistic compared to the performance on truly unseen data, but I'm not aware of any research in that area. However, this suggests a potential research direction - if you try including the dev/test data in the LM pre-training task and find that it significantly improves dev/test accuracy, compared to text that was not included in the LM pre-training task, that would be an interesting approach to improving the performance of language models!

It would be tricky to take advantage of this effect in practice, but you can imagine ways it might be done - for example, at inference time it might be possible to add new inputs to the LM fine-tuning corpus, and then fine-tune your language model on them followed by retraining the classifier (with your pre-existing labelled data) and only then labelling the new inputs! This would probably be too computationally expensive to be used in many production systems, especially when low latency is required, but for some purposes the improvement in data efficiency and accuracy might be worthwhile.<|||||>Great, thank you!

So it's understood that language modeling (or masked LM) is an effective pre-training objective for learning a general representation of the language. But do you think it makes sense to use a MLM head during the fine-tuning phase? For example, if your main task of interest is sentence classification, perhaps you could fine-tune the pre-trained model on a sentence classification head as well as a LM head in a multi-task learning setting. Intuitively it would have a regularizing effect and potentially lead to better generalization. I'd love to hear your thoughts on that idea!<|||||>Take a look at issue #692 , there's a recent Chinese paper where they tried fine-tuning with domain text as well as doing something very similar to what you're proposing (multi-task fine-tuning) and reported good results.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>> 
> 
> Hello,
> 
> I'm thinking about fine-tuning a BERT model using only the Masked LM pre-training objective, and I'd appreciate a bit of guidance. The most straightforward way is probably to modify the simple_lm_finetuning.py script to only do LM fine-tuning. Besides importing BertForMaskedLM instead of BertForPretraining (which has both objectives), what changes should I make, and what potential problems should I consider?
> 
> Also, would it make sense to do MLM fine-tuning on a relatively small domain-specific corpus consisting of just the training examples from the datasets? In other words, does it make sense to do LM pretraining on a corpus that includes training examples taken from the datasets for which you want to make downstream predictions? I'm assuming including the dev/test examples in the corpus is out of the question since it'll likely overfit, but what about just the training examples? I'd like hear to people's thoughts about it.
> 
> Lastly, was the run_lm_finetuning.py script replaced by the contents in the examples/lm_finetuning directory? If so, there are still 2 references to that script in the README that should be edited.
> 
> @Rocketknight1

Hello @dchang56

Did you try fine-tuning a BERT model using only the Masked LM pre-training objective ?

Was that relatively as good as doing that along with next sentence prediction ?

How much  was your block size (maximum sequence length)  for that ?<|||||>> 
> 
> To your first question, the inputs will be almost identical, but the token_type_ids argument will be unused, as this is the vector that indicates the split between the two 'sentences' for the NextSentence objective. I'm not familiar with that part of the code - you might be able to just pass `None` for that argument, or maybe you'll need to pass a vector of zeros that has the right shape (i.e. the same length as the sequence). Also, I believe pre-training with just the masked LM objective works okay - it's by far the most important of the two objectives. The BERT paper didn't show any ablation studies, so I don't know what the effect of removing it will be, but I wouldn't worry too much for a fine-tuning task.
> 
> Secondly, I wouldn't expect a large benefit from just fine-tuning the LM on a small labelled training corpus. The main benefit arises when you have a large corpus to fine-tune the LM on, but only a small fraction of it is labelled.
> 
> Also, I would guess that fine-tuning the LM on the dev/test examples will probably result in test performance that is optimistic compared to the performance on truly unseen data, but I'm not aware of any research in that area. However, this suggests a potential research direction - if you try including the dev/test data in the LM pre-training task and find that it significantly improves dev/test accuracy, compared to text that was not included in the LM pre-training task, that would be an interesting approach to improving the performance of language models!
> 
> It would be tricky to take advantage of this effect in practice, but you can imagine ways it might be done - for example, at inference time it might be possible to add new inputs to the LM fine-tuning corpus, and then fine-tune your language model on them followed by retraining the classifier (with your pre-existing labelled data) and only then labelling the new inputs! This would probably be too computationally expensive to be used in many production systems, especially when low latency is required, but for some purposes the improvement in data efficiency and accuracy might be worthwhile.

Hello @Rocketknight1

why do you think pre-training with just the masked LM objective works okay ?

is there any article or study about that , have you got any link of it ?

Or, have you down any training (BERT) without NSP (next sentence prediction) , yourself ?<|||||>> LM fine-tuning

Could be useful to you: https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/language_modeling.ipynb