How to load multiple TXT training files when pre-train RoBERTa from scratch
https://github.com/huggingface/transformers/blob/d6eeb871706db0d64ab9ffd79f9545d95286b536/examples/pytorch/language-modeling/run_mlm.py#L308

Hi, I want to pre-train RoBERTa from scratch on my dataset. But in example run_mlm.py:
```
        data_files = {}
        if data_args.train_file is not None:
            data_files["train"] = data_args.train_file
            extension = data_args.train_file.split(".")[-1]
        if data_args.validation_file is not None:
            data_files["validation"] = data_args.validation_file
            extension = data_args.validation_file.split(".")[-1]
        if extension == "txt":
            extension = "text"
        raw_datasets = load_dataset(
            extension,
            data_files=data_files,
            cache_dir=model_args.cache_dir,
            use_auth_token=True if model_args.use_auth_token else None,
        )
```
The train file argument `train_file` seems to support only one file:
```
train_file: Optional[str] = field(default=None, metadata={"help": "The input training data file (a text file)."})

data_files["train"] = data_args.train_file
extension = data_args.train_file.split(".")[-1]
```
Because we have too much training data, it is inconvenient to store it in a file. Can train_file support multiple text files?
I would recommend using `datasets` in order to store all of your data; you can then pass it either as a local dataset or as a dataset stored on the Hub to that same script.<|||||>> I would recommend using `datasets` in order to store all of your data; you can then pass it either as a local dataset or as a dataset stored on the Hub to that same script.

Thanks for your reply! How to build a dataset with our data? Is there a tutorial?<|||||>This part of the datasets documentation should likely help out: https://huggingface.co/docs/datasets/loading#local-and-remote-files<|||||>> This part of the datasets documentation should likely help out: https://huggingface.co/docs/datasets/loading#local-and-remote-files

Thanks! It works.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.