As far as I’m concerned, if you need to broadcast the whole dataset to each GPU, why do you even use DDP at the first place? DDP is used for Data Parallel where each GPU get different subset of data. If you need model parallel, you can look at Pytorch rpc.