Hi,
You analysis of when you copy or not is quite accurate.
Some remarks:

If you dataset is of size 100 for example and you want only to work with batches of size 10 and contiguous. You can actually create a Dataset of size 10. Where each idx corresponds to a slice of 10 samples in the true dataset. You can then get these batches randomly with your dataloader.
If you use Dataset/Dataloader, you can use the builtin multiprocessing to load samples from disc asynchronously in multiple processes. See for example the ImageFolder 3 Dataset that loads each images one by one from disk and that is used for datasets like Imagenet. The multiprocessing allows such loader to still be able to feed data fast enough to the gpu to keep it busy.