joshua_mora:

I am using NCCL backend. I do not see how many threads are being passed to create the group. at https://github.com/pytorch/pytorch/blob/master/torch/lib/c10d/ProcessGroupNCCL.cpp
Is there a knob to control that ?


The number of threads is currently not tunable by the user, but we’re considering making this possible in a future release.



 joshua_mora:

I am using dataloder with num_workers set to 1 (so main process spawns a separate process ?)
Based on the information on  https://pytorch.org/docs/1.1.0/_modules/torch/utils/data/dataloader.html.
At each iteration the dataloader process is created and destroyed (if num_workers!=0) which has some overhead ?
Can we keep the processes (depending on how many samples within the batch you want to work concurrently) across iterations so we do not incur into that overhead ?


Right, num_workers=1 would spawn a separate process. Here’s an issue tracking the discussion around keeping subprocesses alive across iterations (with a patch that should make this possible):

  
      github.com/pytorch/pytorch
  
  
    
  
	  
  

  
    
      DataLoader with option to re-use worker processes 12
    

    
      
        opened 
        
          
        
        Jan 9, 2019
      
      


      
        
          
          dashesy
        
      
    
  



  🚀 Feature
Currently after an epoch is ended dataloader spawns a new process to read data. This means if processes have cached...



    feature
    high priority
    module: dataloader
    triaged