Hey all!
I noticed that after the first epoch of training a custom DDP model (1 node, 2 GPUs), two new GPU memory usage entries pop up in nvidia-smi, one for each GPU. A memory usage of around 800 MB is reported for these. Curiously, they have the same PID as the ‘main’ process running on the opposite GPU (see the color-coded screenshot below). Is this expected behavior? It’s the only case of training a model with data parallelism where I noticed it.
I’ve tried using several monitoring tools besides nvidia-smi, e.g. GPUtil.showUtilization(), torch.cuda.memory_allocated(), printing tensors in use, but none of these are descriptive enough for me to pinpoint what exactly is going on.
Any help, suggestions or input would be greatly appreciated!

torch_colored1050×648 373 KB