Do you have any plan on implementing big data files loading functionality?
Suppose I have 300G data files for training, and I can’t load them all into memory.
For now, I am using TensorFlow, and they provide producer/consumer style data loading:
https://www.tensorflow.org/how_tos/reading_data/ 606
With it, I don’t need to read all data set into memory at once, and I can load data in parallel fashion.
Any plan on similar functionality?
Thanks.