Hi,
I have a training scheme where the training data is spread across many sources.
I’ve defined a dataset per source, ending up with many (30-60) datasets (torch.utils.data.ConcatDataset).
Noticed a slowdown behavior watching GPU load during training. Once in a while the GPU load drops to 0 (not periodic).
Tried to look at transforms time Vs. number of workers, but all seems to be working well (when spread across all workers, get_batch time is smaller then net calculations).
All seems to run smooth (GPU well loaded), until number of datasets goes up above a certain number (e.g 30).
Then I experience the non-periodic slowdown.
Thought about memory … but seems like I have enough free memory during the run.
Was hoping one of you could have an idea where should I look next.
Many thanks,
Tal