Hi,
I’ve recently started using the distributed training framework for PyTorch and followed the imagenet 5 example. I’m using multi-node multi-GPU training. While running the code, during the 1st epoch itself, I see multiple processes starting at GPU 0 of both the servers. They are not present initially when I start the training. From the GPU memory usage, it seems that the other processes are some copy of the model (they all have a fixed usage like 571M). Since running an epoch takes ~12 hours for my use case, debugging step by step is not exactly a feasible solution. I’ve ensured that I pass args.gpu  as argument whenever I do a .cuda() call. Also, the model loading/saving is done as suggested in the imagenet example.
Are there any pointers to the probable cause of the issue (or some intelligent ways to debug the code)? Thanks in advance.