Nice work, and I’m also trying to find a efficient way to read large amount of data when training. I think tfrecord format is good way to save data chunks, and avoids reading lots of small files, which is especially slow on hdfs.
In fact, I want to find a so-called TFRecordDatasetLoader which can not only read tfrecord chunks but also have an internal buffer as tf.dataset to provide efficient data reading pipeline.