annisat:

According to the many great threads on this forum, DDP takes care of the synchronization during loss.backward(). But what if the number of data in each data loader leads to different for-loop counts, would the processes with n+1 loops be blocked because the processes with n loops never reach the point?


Yep, the one with n+1 loops will block when using <= PyTorch v1.6. There are ways to get around in user code, e.g. by collecting a signal in each iteration to see if any process has already exited. If yes, break.
@rvarm1 is working on a much better solution, which will be included in v1.7. With that solution, the process that exits early will use dummy comm ops to unblock remaining active ones. Please see the following issue and PR.

  
      github.com/pytorch/pytorch
  
  
    
  
	  
  

  
    
      [RFC] Join-based API to support uneven inputs in DDP 3
    

    
      
        opened 
        
          
        
        May 9, 2020
      
      


      
        
          
          rohan-varma
        
      
    
  



  🚀 Feature
with @pritamdamania87 @mrshenli @zhaojuanmao
This RFC is to summarize the current proposal for supporting uneven inputs across different DDP processes. Related...



    feature
    module: distributed
    triaged


  
  
    
    
  
  



  
      github.com/pytorch/pytorch
  
  
    
  
    
  

  
    
      Join-based API to support DDP uneven inputs 2
    

    
      pytorch:gh/rohan-varma/152/base ← pytorch:gh/rohan-varma/152/head
    

    
      
        opened 
        
          
        
        Aug 5, 2020
      
      

      
        
          
          rohan-varma
        
      

      
        
          +894
          -59