joshua_mora:

Is it possible for example to use same thread for backward and forward if they deal with same model and batch, instead of having 2 threads that could assume using different models and samples ?


I’m not sure of any way to coerce forward and backward into using the same thread.



 joshua_mora:

NCCL process group also accepts the option of size which I am not sure it refers also to the number of threads.


That size actually refers to world_size, which is the total number of ranks in your job



 joshua_mora:

Are there any other hardcoded number of threads that I could reduce ? ( set_num_threads(1)/set_num_interop_threads(1) will not prevent from creating a bunch of threads, larger than 6 per process that deals with each GPU). I have very few cores available per GPU (~4) so I need to restrict the number of threads to what is necessary.


This might provide some more insight into tuning the number of threads: https://github.com/pytorch/pytorch/issues/16894 8. For example, the OMP_NUM_THREADS env var is used for controlling the number of OpenMP threads for CPU operations and MKL_NUM_THREADS for mkl.