Solved by gcramer23 in post #2 
              
              
                @jleguina0 Does DDP fit your need Getting Started with Distributed Data Parallel — PyTorch Tutorials 1.9.0+cu102 documentation? You will have one model replica per process. Instead of storing the gradients in a shared model DDP will update all the model replicas. You can define a communication hook D…