I’m trying to build a simple CNN where the input is a list of NumPy arrays and the target is a list of real numbers (regression problem).
I’m stuck when I try to create the DataLoader.
Suppose Xp_train and yp_train are two Python lists that contain NumPy arrays. Currently I’m using the following code:
tensor_Xp_train = torch.stack([torch.Tensor(el) for el in Xp_train])
tensor_yp_train = torch.stack([torch.Tensor(el) for el in yp_train])
dataset_p_train = TensorDataset(tensor_Xp_train, tensor_yp_train)
loader_p_train = DataLoader(dataset_p_train)

The problem is that it’s not working since Xp_train is a list of 149798 arrays.
Is there a way to create a DataLoader without loading everything into memory?