Solved by mrshenli in post #6 
              
              
                here is the link: DistributedDataParallel — PyTorch master documentation 
no_sync basically disables DDP comm within the context.