Efficient data loading functionality
## 🚀 Feature

Efficient data loader for huge dataset with lazy loading! 

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request related to a problem? e.g., I'm always frustrated when [...]. If this is related to another GitHub issue, please link here too. -->

I am working with a huge dataset consisting of 120m examples (~40G raw text) in a single csv file. I tried to follow the [run_glue](https://github.com/huggingface/pytorch-transformers/blob/master/examples/run_glue.py) distributed training example, however this is too slow as it first creates all the examples and cache it [here](https://github.com/huggingface/pytorch-transformers/blob/master/examples/run_glue.py#L251). Basically only first process in distributed training process the dataset and others just use the cache. 

Is there any data loader (or a working example) that would be efficient for training the model on such a huge dataset? 

## Additional context

<!-- Add any other context or screenshots about the feature request here. -->
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>@thomwolf Is there an efficient implementation for this? Could you re-open the issue please. <|||||>CSV generally has slower load times. How about benchmarking with pickle, parquet, and feather? Pytorch's dataloader can handle multiple files and multiple lines per file 
https://discuss.pytorch.org/t/dataloaders-multiple-files-and-multiple-rows-per-column-with-lazy-evaluation/11769 