Hi, the issue here turned out to be with padding, so apologies with the misleading original post. I thought we had investigated it fully but we had not. I’ve added more of a description with some possible helpful notes here:
  
      github.com/huggingface/transformers
  
  
    
  
	  
  

  
    
      DDP performing slightly worse in terms of loss and metrics than DP 45
    

    
      
        opened 
        
          
        
        Feb 24, 2021
      
      

        
          closed 
        
          
        
        Mar 16, 2021
      
        

      
        
          
          johncookds
        
      
    
  



  Hi,
I am running Transformers 4.3.2 and am testing DistributedDataParallel vs DataParallel. The Huggingface Trainer in both instances have been untouched and...