Solved by wayi in post #2 
              
              
                Is it possible to disable this functionality so I can call all reduce manually? 

Do you need to implement any customized logic in allreduce? If so, I will recommend DDP comm hooks, which provides an interface to implement customized allreduce. 
Another option is no_sync context manager, which will…