Dataset Lazyloader for transformers trainer
# 🚀 Feature request

Currently, train/fine-tune models using transformers trainer loads the whole dataset in memory.
This could work for small datasets, but with large datasets the training will crash.

Code:
https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_language_modeling.py#L128

## Motivation

We have trained 0.4 Billion Bert Model and currently training a 1.8 Billion Bert Model for protein sequences, and we want to train DistilBERT to reduce the model size.

Unfortunately, our dataset is very huge about 0.7 Terabyte and since the trainer loads the whole dataset the trainer crashes.

It will be more optimised if you could use lazy loader for loading the datasets like fairseq.

@patrickvonplaten  I am not sure who is responsible for the data loader for LM.
As far as I can see there is currently no dataloader functionality to lazy load data into memory. 
It should not be very hard to implement such a feature yourself, I think, see https://discuss.pytorch.org/t/loading-huge-data-functionality/346/2 .
Also cc @lhoestq @sgugger @thomwolf - Are we planning on providing more support for lazy data loading? <|||||>There is an [open PR #4009](https://github.com/huggingface/transformers/pull/4009) about this that was mostly close to being merged (cc @BramVanroy) and I must confess I kind of dropped the ball on it.

Shouldn't be a ton of work to complete it and get it merged.

As mentioned in that PR, the other option is to use `huggingface/nlp` which can load large text datasets lazily out of the box.<|||||>@patrickvonplaten @julien-c Thanks a lot for your reply.

Should we close this issue and focus on https://github.com/huggingface/transformers/pull/4009  ?<|||||>The long-term solution is to use nlp for this IMO.<|||||>I have checked NLP and it is slow, maybe I am doing something wrong.
I made a simple python script to check it is speed, which loads 1.1 TB of textual data.
It has been 8 hours and still, it is on the loading steps.
It does work when the text dataset size is small about  1 GB, but it doesn't scale.
It also uses a single thread during the data loading step.

```
train_files = glob.glob("xxx/*.txt",recursive=True)
random.shuffle(train_files)

print(train_files)

dataset = nlp.load_dataset('text', 
                           data_files=train_files,
                           name="customDataset",
                           version="1.0.0",
                           cache_dir="xxx/nlp")
```<|||||>You should open this issue on the nlp repo, to make sure it's not forgotten! In particular they are working a lot on performance right now :-)<|||||>done:
https://github.com/huggingface/nlp/issues/546
<|||||>I am closing this in favour of:
- https://github.com/huggingface/transformers/pull/4009
- https://github.com/huggingface/nlp/issues/546
