command line_by_line missing in https://github.com/huggingface/transformers/tree/master/examples/language-modeling
## Environment info
- `transformers` version: 3.5.0
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.6
- PyTorch version (GPU?): 1.7.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help
@LysandreJik @patrickvonplaten @TevenLeScao


## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [ X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)
https://github.com/huggingface/transformers/tree/master/examples/language-modeling


## To reproduce

In the old version of the script ``run_clm.py`` called ``run_language_modeling`` there was an argument ``line_by_line`` which allowed to read the data by putting each sequence on a line. This argument seems to be missing in the newer version 
``run_clm.py``. 

## Expected behavior
Perhaps there is an argument that has replaced ``line_by_line``  but I don't really see that. Sorry if I missed something. 


Pinging @sgugger here - think he knows `run_clm.py` best<|||||>`run_clm` does not have the `line_by_line` option as it doesn't make sense for causal language modeling: pretraining for causal language modeling is done by concatenating all available texts separated by a special token, then building sequences of a certain `block_size` with them. Using `line_by_line` and keeping the sentences separate result in the model having to predict the padding token quite often (which pretrained causal models usually don't have) and without knowing when to stop predicting that padding token.

Only `run_mlm` keeps that option as it makes sense to have sentences of different lengths for masked language modeling. You can always copy the relevant bit of code in `run_mlm` to use it in `run_clm` but I would strongly advise against it.<|||||>Thank you for your reply, it makes sense. So that means that I need to give the script the concatenated data (and separate sequences by a special token)? Or does the script ``run_clm`` take care of that? <|||||>The script takes care of that for you :-)<|||||>This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.<|||||>@sgugger sorry to revive that thread.
I think a clm with two special tokens BOS / EOS would make sense to be trained in line by line mode, what do you think ?
(btw are you saying that all pre-trained gpt2 models are trained in fixed blocks ? if so do you confirm that original papers do the same when benchmarking with standards like 1BW ?)
thanks for your insight.