Edit: Mistaken!
This was my issue:

  
      github.com/pytorch/pytorch
  
  
    
  
	  
  

  
    
      DistributedDataParallel: resume training from a checkpoint results in additional processes on GPU 0 45
    

    
      
        opened 
        
          
        
        Jul 21, 2019
      
      

        
          closed 
        
          
        
        Jul 21, 2019
      
        

      
        
          
          qchenclaire
        
      
    
  



  Hi,
When I was trying the imagenet example with DistributedDataParallel, using single node with 4 gpus, I found that when I add...



    module: distributed


  
  
    
    
  
  



13%20PM1416×458 32.4 KB

Here’s a screenshot of distributed training in Pytorch when I call the train function like:
CUDA_VISIBLE_DEVICES=1,2,3,4 python -m torch.distributed.launch --nproc_per_node=4 train_new.py. You can see that the first rank has also initted 3 separate processes for each other GPU. When I use 10 GPUs on a box this severely limits the batch size, since the 0th dimension node has so much less capacity. What is it storing? I thought gradients in DDP were all-reduced. I’ve also tried turning broadcast_buffers to False to no avail.
Model is stacked modules of 1D-conv, relu, batch norm, LSTM, followed by a large softmax layer and CTC loss. Backend is NCCL
Pytorch 1.3.0, Cuda 10.1, Titan RTX, Ubuntu 18.04. Can provide more code upon request.