Hi All,
I’m trying to create a DataSet class that can load many large file, and each file have rows of data a model would need to train.
I’ve read: Loading huge data functionality
class MyDataset(torch.utils.Dataset):
    def __init__(self):
        self.data_files = os.listdir('data_dir')
        sort(self.data_files)

    def __getindex__(self, idx):
        return load_file(self.data_files[idx])

    def __len__(self):
        return len(self.data_files)


dset = MyDataset()
loader = torch.utils.DataLoader(dset, num_workers=8)

Essentially if each file is an image, then you can use this functionality to only load the required images in memory.
So what about a file with multiple lines? Then I came across this github thread: https://github.com/pytorch/text/issues/130 130
class LazyTextDataset(Dataset):
    def __init__(self, filename):
        self._filename = filename
        self._total_data = 0
        with open(filename, "r") as f:
            self._total_data = len(f.readlines()) - 1

    def __getitem__(self, idx):
        line = linecache.getline(self._filename, idx + 1)
        csv_line = csv.reader([line])
        .....

    def __len__(self):
        return self._total_data

Now the problem I have is how do I combine these two so that I can load multiple files that are all large?
Since __getitem__ only uses 1 idx, there isn’t a way to load multiple file and each of their lines?
Can someone point me to the right direction?