This sounds like a valid approach.
Alternatively, you could of course create a tensor or list with all slices and use the index to grab a slice and create the desired batch from it, but I guess load_everything_in_memory might then become a bit more complicated.
You could try to use torch.utils.checkpoint to trade compute for memory, but I’m not sure how large your memory requirement is for a single batch.