@DanielWang Thanks for posting the question. Could you elaborate more on what’s your use case? from the description I see you have two MLP models with exactly the same architecture, and trained separately by two parallel process, it looks to me you can just use DDP with the MLP model, and it will periodically (full_sync or async) sync the model parameters gradients by DDP under the hood? Getting Started with Distributed Data Parallel — PyTorch Tutorials 1.9.1+cu102 documentation