IIUC, DistributedDataParallel does not fit in this use case because you have a population of independent models to train on the same set of data instead one big model on different splits of input data. It looks like the experimental torch.distributed.rpc 31 might be helpful here, it would at least help you take care of the communication. But you would still need to write code to dispatch models to the workers in the pool.