Yes, thanks for your mention, I also have problems with data persistence.


Split data into train_set and test_set
Previously I loaded all data into memory and random index it to generate train_set and test_set, but right now I don’t know how to do this better. The only way I know is to split .csv file in advance then process them by chunking, is there any better way?


Store the processed data
The data processing procedure will expand data X20-30 times so it’s better to dump them. Is there any recommended way/format to do this? I guess I can do this in “”“getitem”"" function.