Thanks for the pointer to this discussion on the dataloader.
With respect to the number of threads/processes, I still miss to understand all the other threads being generated.
Is it possible for example to use same thread for backward and forward if they deal with same model and batch, instead of having 2 threads that could assume using different models and samples ?
NCCL process group also accepts the option of size which I am not sure it refers also to the number of threads.
Are there any other hardcoded number of threads that I could reduce ? ( set_num_threads(1)/set_num_interop_threads(1) will not prevent from creating a bunch of threads, larger than 6 per process that deals with each GPU). I have very few cores available per GPU (~4) so I need to restrict the number of threads to what is necessary.
Thanks, again.