SortedDL for contiguous LM
Hi there,
I am currently implementing LM re-training of a RoBERTa model using the `Trainer` API. Since I have a huge training corpus, I was wondering if there is a functionality in the `Trainer` or the corresponding `DataCollatorForLanguageModeling` that allows for sorted batching as in `fastai`? 

More precisely, I would like to feed in all my training data as a contiguous text stream and let the respective functions handle sorted batching irrespective of the sequence length of the individual sequences.

Best,
Simon
This issue is more suited for the forum, but maybe @sgugger has some hints to share!<|||||>I'm not sure what you would want to sort a single text stream. The `Trainer` supports `--group_by_length` but that's when you have multiple texts.

Note that `DataCollatorForLanguageModeling` only performs random masking on your prepared data, nothing more.<|||||>Sorry, maybe I was imprecise here (still trying to wrap my head around a lot of these new concepts). Essentially, what I was wondering if batches for language modeling are constructed in a way that is similar to the approach that you described in [Chapter 10: 
NLP Deep Dive](https://github.com/fastai/fastbook/blob/master/10_nlp.ipynb) (section 'Putting Our Texts into Batches for a Language Model') of the book you co-authored. If I got it correctly, that would allow me to train the transformer over variable length input sequences without worrying about sequences being truncated due to the constraints imposed by `tokenizer.max_len_single_sentence` (overflowing parts would simply end up at the appropriate position in the nex mini-batch).<|||||>I think you may be referring to the LM DataLoader. This kind of preprocessing is done using the `datasets` library on our side. Take a look at the [run_clm](https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_clm.py) or [run_mlm](https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm.py) examples (in run_mlm the part that is not in the block "line_by_line") or the [language modeling notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/language_modeling.ipynb) to see how.<|||||>Thanks a lot! Indeed, the `group_text()` function was exactly what I was looking for. <|||||>Hey @sgugger,

sorry for reopening this one. I am still not 100% sure if I conveyed my issue properly the first time. Hence, let me briefly restate it using an example:

Let's say I chunk my text stream of length 100 (let's say including 3 sentences) into blocks of `block_size=5`, resulting in 20 blocks. Now, I'd like to feed them into my model using a `batch_size=5`, resulting in 4 batches à 5 text blocks.

I am still not 100% sure how they are fed into the model using the `DataCollatorForLanguageModeling` and `Trainer` API:
```python
# Variant A

batch_1 = [block1, block2, block3, block4, block5]
# ...
batch_4 = [block16, block17, block18, block19, block20]
``` 

```python
# Variant B

batch_1 = [block1, block5, block9, block13, block17]
batch_2 = [block2, block6, block10, block14, block18]
# ...
batch_4 = [block4, block8, block12, block16, block20]
``` 

If I got it correctly, the method presented in the book referred to in the previous comment relates to *variant B*.<|||||>You will need to write your own data collator for that as this is not in the Transformers library: contrary to LSTMs, Transformers do not have a state so we don't care about the ordering across batches for those models.<|||||>This makes entirely sense, thanks for lifting this barrier in my head! 