Hi,
Could you maybe add a source code with reproduction of the problem?
One more suggestion. I’ve recently looked at a problem

  
      github.com/pytorch/pytorch
  
  
    
  
	  
  

  
    
      Wrong gradients when using DistributedDataParallel and autograd.grad. 62
    

    
      
        opened 
        
          
        
        Nov 7, 2020
      
      

        
          closed 
        
          
        
        Nov 19, 2020
      
        

      
        
          
          TropComplique
        
      
    
  



  🐛 Bug
The gradient reduction across workers doesn't work when:
gradient penalty is used as a loss,
a bias is used in the last...



    oncall: distributed


  
  
    
    
  
  


where loss function didn’t depend on all the parameters and it caused some issues for the gradient computation. What helped is to look at the autograd graph via https://github.com/szagoruyko/pytorchviz 42
Could you create such a graph from your model?