Wrote a blog about a way to use all_gather, without the need to calculate the gradient

  
      Medium – 7 Feb 21
  
  
    

Gradient backpropagation with torch.distributed.all_gather 136

This blog presents some tips on how to use torch.distributed.all_gather and make sure the gradient is correctly calculated for deep…

  Reading time: 4 min read