I have some data which is thrice large as my system’s RAM. I need to run some Deep Learning models using pytorch. Could you please advise how can I use torch data loaders (or alternative) in this scenario?
Assume my data is stored as subfolders inside parent directory as below.
Transaction_Data/
              ---Customer1/
                          --- day1
                          --- day2
                          .
                          --- dayN
              ---Customer2/
                          --- day1
                          --- day2
                          .
                          --- dayN
             ---CustomerN/
                          --- day1
                          --- day2
                          .
                          --- dayN

Lets assume these are clean data and each customer is like an individual data frame (days represent rows).
I want to load these data in batches (probably 5 customers at once - can fit into memory) and train DL model using torch. What is the efficient way to load these?
I need to iterate over these data and run DL models.  Should I be using custom dataset? Any pointers would be really appreciated.
Thank you very much in advance!