Fine-tuning on Language Model using two tasks
Hi,

I'm reading the language modeling example on the documentation: https://huggingface.co/transformers/v2.0.0/examples.html#language-model-fine-tuning

It seems that the fine-tuning is done based on Masked Language Modelling (MLM) loss. While in the BERT paper, it appears that the LM fine-tuning is done by optimizing two tasks: 1) Masked Language Modeling, and 2) Next sentence prediction. I'm looking for the second part in Huggingface's implementation, but it seems that this part is either not implemented or I'm missing something? 
Hi, you're not missing anything, this part is not implemented in the examples, as fine-tuning a model using only MLM yields similar downstream results than fine-tuning a model with both tasks. 

However, we have the `BertForPreTraining` architecture which is implemented, and which can train a model using the two objectives. You would have to tweak the example scripts to manage this case, however.<|||||>Also, we try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discusss.huggingface.co) instead?

Thanks!<|||||>> Also, we try to keep the github issues for bugs/feature requests.
> Could you ask your question on the [forum](https://discusss.huggingface.co) instead?
> 
> Thanks!

Sure, thank you for clarifying!