Input file format for examples/run_lm_finetuning.py
# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->

I wanted to employ the `examples/run_lm_finetuning.py` from the transformers repository on a pretrained Bert model. However, from following the documentation it is not evident how a corpus file should be structured (apart from referencing the Wiki-2 dataset). I've tried

- One document per line (multiple sentences)
- One sentence per line. Documents are separated by a blank line (this I found in some older pytorch-transformers documentation)

By looking at the code of `examples/run_lm_finetuning.py` it is not directly evident how sequence pairs for the Next Sentence Prediction objective are formed. Would the --line-by-line option help here? I'd be grateful, if someone could give me some hints how a text corpus file should look like.

Many thanks and cheers,

nminds

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**: 
[SO link](https://stackoverflow.com/questions/60001698/how-exactly-should-the-input-file-be-formatted-for-the-language-model-finetuning)
Hi, two datasets are available in `run_lm_finetuning.py`:

- `TextDataset`, which just splits your data into chunks with no attention whatsoever to the line returns or separators
- `LineByLineTextDataset`, which splits your data into chunks, being careful not to overstep line returns as each line is interpreted as a document.

None of those datasets, nor the run_lm_finetuning script in itself handle the next sentence prediction objective. It handles masked language modeling when the `--mlm` flag option is passed, and the causal language modeling when no `--mlm` flag option is passed.<|||||>@LysandreJik 
[this](https://github.com/huggingface/transformers/blob/33d3072e1c54bcd235447b98c6dea1b4cb71234c/examples/run_lm_finetuning.py#L135) will drop tokens beyond len of `512`?<|||||>> Hi, two datasets are available in `run_lm_finetuning.py`:
> 
> * `TextDataset`, which just splits your data into chunks with no attention whatsoever to the line returns or separators
> * `LineByLineTextDataset`, which splits your data into chunks, being careful not to overstep line returns as each line is interpreted as a document.
> 
> None of those datasets, nor the run_lm_finetuning script in itself handle the next sentence prediction objective. It handles masked language modeling when the `--mlm` flag option is passed, and the causal language modeling when no `--mlm` flag option is passed.

Hi, Is there any particular reason to exclude the next sentence prediction objective? <|||||>@nauman-chaudhary it will drop the tokens beyond the maximum input size of the model. For BERT, it is indeed 512. Feel free to implement a more complex behavior if your dataset has a lot lines that go over the 512 token limit.

@fajri91  Yes, for a couple of reasons:

- Having a simple MLM/CLM objective is simpler, both to understand (user) and to maintain (maintainer)
- The RoBERTa paper has proven that the NSP objective was not particularly helpful
- Only BERT has the class (`BertForPreTraining`) to manage the NSP objective, whereas `run_lm_finetuning` supports several models available in the library
- If anyone wants to implement the NSP objective, it is very easy for them to change the dataset/training loop to do so.<|||||>@LysandreJik given the fact that tokenizer drops input size above 512 is it worth to prepare the input dataset by using sliding window over documents? What I mean by that is instead of dropping a lot of text, I will transform long document into i.e 4 sentences in each line, with sliding window over the whole document.<|||||>Yes, this is a reasonable strategy.<|||||>Thanks for clarification, it's super helpful to know this!<|||||>> Hi, two datasets are available in `run_lm_finetuning.py`:
> 
> * `TextDataset`, which just splits your data into chunks with no attention whatsoever to the line returns or separators
> * `LineByLineTextDataset`, which splits your data into chunks, being careful not to overstep line returns as each line is interpreted as a document.
> 
> None of those datasets, nor the run_lm_finetuning script in itself handle the next sentence prediction objective. It handles masked language modeling when the `--mlm` flag option is passed, and the causal language modeling when no `--mlm` flag option is passed.


hello @LysandreJik 

if i realized TextDataset correctly  , it  makes a long sentence of all the corpus and cut them 512 by 512  and gives it to the model (if the max_len is supposed to be 512 ) and then we will have no padding

but  LineByLineTextDataset ,  pad each line to reach 512 and gives it to model 

by which one we will get better results in downstream tasks ?
(and in downstream tasks we are unforced to do padding) 

thanks!<|||||>@marrrcin @LysandreJik thank you for the comments! I  have a domain specific corpus; Geology, which is about 2GB text file. I prepared an input file where, I scanned a 4-sentences window on my raw text, and wrote each 4-sentence window onto a new line.  

So, my modified input file has 4 sentences per line ending with \n.   Now I am training BERT MLM training with BertWordPieceTokenizer from scratch.  run_language_model.py gets to LineByLineTextDataSet and takes almost 2 hours to process my input file.  I feel this is quite slow for only 2GB file.

My input looks something like this, there is no space between lines and there are 2 millions of lines:
{
first sentence starts here. second sentence. now third sentence. and forth sentence
new line with fifth sentence. sixth sentence here, then seventh sentence. finally eight sentence
new line with night sentence, then tenth sentence ...
...
}

is there anyway to speed this up? <|||||>just upgraded HF to v.2.9 and now it took 37 minutes instead of 108 minutes. thanks!<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
