Ah thank you,
I think I miscommunicated. We are not moving to a truly distributed setup. I was more curious as to the distribution of the data loading when we specify num_workers amongst the workers. But I took a look at the source code and think I understand.
Your first post helped a lot and we have the dynamic dataloader working.
Thank You!