So i read this thread and some other, googled and etc

  
    
    
    Training performance degrades with DistributedDataParallel distributed
  
  
    I’m training a conv model using DataParallel (DP) and DistributedDataParallel (DDP) modes. For DDP, I only use it on a single node and each process is one GPU. 
My model has many BatchNorm2d layers. Given all other things the same, I observe that DP trains better than DDP (in classification accuracy). Even if I add SyncBN from pytorch 1.1, I still observe that DP > DDP+SyncBN > DDP without SyncBN in test accuracy. 
I’m aware of the difference between DP and DDP’s handling of averaging/sum: Is av…
  


My model is seq to seq variable len input (tacotron 2).
This is how looks like loss
                    
            
           6


Any suggestions? Any way how to debug it?
Also if i remove all batchnorm from model, it raise unused variable error, how it work?