What I ended up doing that worked quite well was split the data between files with training samples of 1000 each quite easily by setting the filename to 'data-'+str(i//1000)+'.csv'. Since the data has 180 points of data per training sample, the csv file sizes are just a couple mb each. But if your training samples are larger, the same would work by changing 1000---->100.
Just make sure to print “i” at the end of your preprocessing script just to make sure you know what to set the __len__ to in your CustomDataset.