Loading a set of tokenized files for training
I have a directory of files that are already tokenized using a pretrained tokenizer. Each file is a pickle file containing a list of objects where each object corresponds to a text sequence containing input_ids and attention_masks. The directory has thousands of files. I'm looking for an efficient way to load the data for training using Trainer. Do I have to write my own Dataloader or do I create a custom dataset using Datasets? 

Thank you. 
In this case, you could write a custom `dataset` that will read your pickle files and return the examples from `__getitem__` method.

If you are looking for an efficient way of pre-tokenizing the dataset, saving/caching it for future use, and loading it for training then I would recommend you to take a look at [datasets](https://github.com/huggingface/datasets) library. It takes care of caching your pre-processed data and loading it efficiently (lazy loading, so memory won't blow up)<|||||>Thanks Suraj <|||||>This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.