Awesome!! And yes, let’s wait for comments from @Kiuk_Chung
If this looks OK, we can then start from publishing an RFC design issue. Here are some examples:

  
      github.com/pytorch/pytorch
  
  
    
  
	  
  

  
    
      [RFC] RPC Based Distributed Model Parallel 1
    

    
      
        opened 
        
          
        
        Jul 19, 2019
      
      


      
        
          
          mrshenli
        
      
    
  



  with @pritamdamania87 @zhaojuanmao @aazzolini @gqchen @pietern @satgera @ezyang @zdevito @suo @manojkris @gchanan @soumith @dzhulgakov @yifuwang @bddppq @joxu-cn @dwarakrajagopal @jspisak
PyTorch currently provides...



    feature
    module: rpc
    triaged


  
  
    
    
  
  



  
      github.com/pytorch/pytorch
  
  
    
  
	  
  

  
    
      [RFC] Join-based API to support uneven inputs in DDP 1
    

    
      
        opened 
        
          
        
        May 9, 2020
      
      


      
        
          
          rohan-varma
        
      
    
  



  🚀 Feature
with @pritamdamania87 @mrshenli @zhaojuanmao
This RFC is to summarize the current proposal for supporting uneven inputs across different DDP processes. Related...



    feature
    module: distributed
    triaged


  
  
    
    
  
  



  
      github.com/pytorch/pytorch
  
  
    
  
	  
  

  
    
      [RFC] Async User Function for RPC 2
    

    
      
        opened 
        
          
        
        Apr 6, 2020
      
      


      
        
          
          mrshenli
        
      
    
  



  Impact
Currently, every RPC request occupies an RPC thread on the server side until done. However, if there are nested RPC calls...



    feature
    module: rpc
    triaged


  
  
    
    
  
  



  
      github.com/pytorch/pytorch
  
  
    
  
	  
  

  
    
      [RFC] DDP Communication Hook
    

    
      
        opened 
        
          
        
        May 29, 2020
      
      


      
        
          
          pritamdamania87
        
      
    
  



  Motivation
There are several algorithms like GossipGrad and gradient compression which involve different communication strategies for parameter syncs while running Distributed DataParallel...



    enhancement
    module: distributed
    triaged


  
  
    
    
  
  


This will draw people into the discussion.