Hi Mike @pgmmpk ,
Thanks for posting your solution!
Did you find a way to integrate tfrecord iterator with PyTorch DataLoader?
Also, @ptrblck - is there a tfreocrd-like solution in PyTorch? Essentially, I would like to handle big datasets the same way tfrecord-based dataset works - read big files and shuffle the serialised samples (like tf example protobuf) rather than reading batch_size images in every step, which results in I/O overhead.
Another issue is that our file system in which we store datasets is having trouble handling so many files, so this is another advantage for tfreocrd-based datasets.
Thank you!!
Adva