Hi @Blake_Camp Thanks for posting the question. You can try mixing DDP and RPC together, use RPC to hold a shared part of your model in one rank, and use DDP for the remaining part. see Combining Distributed DataParallel with Distributed RPC Framework — PyTorch Tutorials 1.9.1+cu102 documentation 1