Thanks for the quick reply.
I am using NCCL backend. I do not see how many threads are being passed to create the group. at https://github.com/pytorch/pytorch/blob/master/torch/lib/c10d/ProcessGroupNCCL.cpp 3
Is there a knob to control that ?
I am using dataloder with num_workers set to 1 (so main process spawns a separate process ?)
Based on the information on  https://pytorch.org/docs/1.1.0/_modules/torch/utils/data/dataloader.html 3.
At each iteration the dataloader process is created and destroyed (if num_workers!=0) which has some overhead ?
Can we keep the processes (depending on how many samples within the batch you want to work concurrently) across iterations so we do not incur into that overhead ?
I am basically trying to prune the number of processes and threads, while I understand I may restrict generality but I am trying to speed up the execution when I am CPU bound.
Thanks in advance.