Solved by albanD in post #2 
              
              
                The DDP() wrapper takes care of all the synchronizations and offer a nn.Module like api so that you can use it transparently.