This is really intriguing.
Did you get the chance to benchmark your implementation ? @pgmmpk
Specifically to read this huge dataset : https://research.google.com/youtube8m/download.html 24
(1.53 tb of tfrecords)
What would be an optimum approach ?

running a tf sess dataloader
Using the above implementation
saving as framework agnostic npy etc and then re reading into a pytorch dataloader ?

@ptrblck