Thanks for the fast reply!
The tfreocrd mechanism designed to handle large datasets. It based on protobuf serialisation protocol that is used for creating training “examples”. For instance, an “example” can be composed of training image and an integer label. All the examples are serialised and written to a tfrecord file (or files). This way, you can compress a dataset like ImageNet to only ~1000 records.
The benefits are: (1) file system performance since it doesn’t have to handle millions of files, (2) training performance since instead of reading a lot of small files the training-process reads few big files, and (3) dataset mobility - transfer few big files instead of many small files.
These datasets usually don’t fit the CPU memory, so I can’t load the complete dataset before training.