def read(self):
        self.read_file = True
        for ep in self.episode_lst:
            print(ep)
            filename = self.file_dict[ep]
            tup = torch.load( filename )
            print('Finished loading')
            if self.downsample == 0:
                self.file_dict[ep] = tup
            else:
                tup_new = (
                    tup[0][:,::self.downsample], 
                    tup[1][:,::self.downsample]
                    )
                self.file_dict[ep] = tup_new
        return self

This is the code snippet for loading files in my dataset object. self.episode_list is a list of IDs for the individual files, self.file_dict is a dict containing the path to each individual file. When I’m using a subset of the dataset for preliminary results, I can fit the whole dataset into memory, and therefore will directly load the files into the dataset object then create DataLoader objects, instead of loading files in DataLoader.