@Hongkai_Zheng ,
Thank you!
I will cooking up a toy example to validate my idea and understanding.



 Hongkai_Zheng:

Regarding the code you referred to, it uses DistributedSampler (check createdataset function in the utils.dataset.py ) to divide dataset into several subsets so that each GPU only sees one subset of the whole dataset.


Do you think the code is necessary to broadcast the dataset to each GPU?
or in what case master process should broadcast dataset to each GPU?