Single sentence corpus in run_lm_finetuning?
Hi,

I am trying to pre-train using `BertForPreTraning` in `run_lm_finetuning.py`. My target corpus is based on very many tweets and I am unsure how the model will tackle that since they are mostly only one sentence. Will it affect the IsNextSentence task? 

Should my .txt input file consist of one tweet on each line where each tweet is seperated by an empty line?
https://github.com/huggingface/pytorch-pretrained-BERT/issues/272

I had the same issue and but apparently this cant be done in BERT<|||||>Yes, can't be done currently. Feel free to submit a PR to extend the `run_lm_finetuning` example @vebits!