Hi,
Suppose I have a folder which contain multiple files, Is there some way for create a dataloader to read the files? For example, after a spark or a mapreduce job, the outputs in a folder is like
part-00000
part-00001
...
part-00999

Usually the files in the folder is very large and cannot fit to memory.
For TensorFlow 2.0, we can convert the file to tfrecord format and feed the folder path TFRecordDataset to read the data smoothly and efficiently, like below
raw_dataset = tf.data.TFRecordDataset(folder_path)
def _parse_example(example_string): 
    feature_dict = tf.io.parse_single_example(example_string, features)
    return feature_dict
dataset = raw_dataset.map(_parse_example)

However, I did not find a proper way for pyTorch to deal with this situation. I have searched the answer on the internet for a long time but I did not figure it out.
So I come here to get some help.
Actually, I think this is a normal requirements, I would like to have the answer. And I believe the powerful PyTorch should have an elegant way to this problem, like Tensorflow 2.0.
Thanks very much!