You can implement a custom dataset that does lazy loading by writing the data loader as a Python generator (with yield) and using an iterator that doesn’t sort or shuffle the data except within small buckets. That way the iterator will request a bucket worth of data, the loader will load it into memory, and the iterator will sort/shuffle it without loading the whole dataset.