Hey @sakh251
I just wanna mention one relevant WIP project that we are working on. We are running internal reviews on the code-level design on it, and should be able to share on Github in the next few weeks. This should help make customizing DDP a lot easier.

  
      github.com/pytorch/pytorch
  
  
    
  
	  
  

  
    
      [RFC] Modularize DistributedDataParallel 1
    

    
      
        opened 
        
          
        
        Apr 21, 2020
      
      


      
        
          
          mrshenli
        
      
    
  



  Summary
This project aims at decomposing existing DistributedDataParallel (DDP) implementation into multiple smaller pluggable and customizable building blocks. So that applications can...



    feature
    oncall: distributed
    triaged