I’m not really familiar with tfrecord, but from your description it sounds like the whole dataset is being loaded into the memory first and then just the current sample is loaded. Is that right?
If so, you could load the data once and save it using torch.save. Afterwards, you could load it in your Dataset's __init__ and get a sample in __getitem__.
Please correct me, if I misunderstood the tfrecord approach.