Sorry, I’m late to the party. Have you tried serving your data from a TensorDataset with mmapped tensors? Then initial load time goes to zero because the tensor is never actually loaded into memory as a whole. Something like:
torch.FloatTensor(torch.FloatStorage.from_file("frames.bin", True, -1)).view(-1, 3, 224, 224)

To help with disk seeking, write a scheduler that would keep skipping forward your dataset (and then wrap around) with relatively small but random steps. Take the steps from a one-sided Cauchy or Student’s t distribution to get a bunch of small steps with a few large skips.