I think that the model’s parameter tensors will have their data moved to shared memory as per Multiprocessing best practices — PyTorch 1.9.0 documentation 4, so you’d essentially be doing Hogwild training and this could cause issues with DistributedDataParallel as usually the model is instantiated individually on each rank. Is there a reason you can’t simply create the model within each subprocess?
Regarding the question about the data loader, I think if the dataloader is serializable this should work, but it would result in each worker training on the same data, and thus DDP would not really help as you’re using the same data and different model. Is it possible for you to do something like pass in a file handle/file name to each worker, and possibly an index/offset so you can shard your data across multiple workers?