Yes, that’s more or less what happens.
After some isolated tests of just loading batches from the dataloader without doing any GPU processing, I think the problem could stem from the size of the epoch and how the dataloader iterates over epochs. The current dataset only has 50 samples (long videos) from which smaller samples are taken. The dataloader iterates over epochs of 50. Is it possible that the dataloader only prefetches until the end of the epoch is reached and after the last batch is prefetched, it waits until completion of all batches on the GPU? Afterwards it somehow “restarts”? This could explain the behavior I’ve seen lately: Near 100% GPU utilization with regular intermediate dips to 0%
(I’m using a map-style dataset)
EDIT:
Probably that’s the issue right here (https://pytorch.org/docs/stable/data.html 8):
Multi-process data loading
Setting the argument  num_workers  as a positive integer will turn on multi-process data loading with the specified number of loader worker processes.
In this mode, each time an iterator of a  DataLoader  2 is created (e.g., when you call  enumerate(dataloader) ),  num_workers  worker processes are created. At this point, the  dataset ,  collate_fn , and  worker_init_fn  are passed to each worker, where they are used to initialize, and fetch data. This means that dataset access together with its internal IO, transforms (including  collate_fn ) runs in the worker process.
Shouldn’t the implementation of torch.utils.data.DataLoader take care of seamless prefetching over epochs?