Could you use the DataLoader 1 class with a custom dataset? It has options for controlling prefetching, memory pinning to speed up moving data to and from the GPU, and multiple workers if the loading can be parallelised.