This is just an example. I am actually using a for loop that iterates over all the csv files. Each column in the file is a datapoint and in the end (self.data[i]), I am returning the column because that’s what I want batches of. This is how I’m implementing the TrainLoader:
train_loader = torch.utils.data.DataLoader(
         train_dl_spec, batch_size=128, shuffle=True, num_workers=8, pin_memory=True)
for data in train_loader:

This is how I am implementing it in the training process:
for x_train in tqdm(train_files):
        train_dl_spec = data_gen(x_train)
        train_loader = torch.utils.data.DataLoader(
        train_dl_spec, batch_size=128, shuffle=True, num_workers=8, pin_memory=True)
        for data in train_loader: