The 2 figures are from two machine with same GPU, driver, pytorch and code (perhaps the batch size is different). But the gpu processes are different. The number of  “small” process in first figure changed with num_workers in dataloader. The main problem is that the training program in first figure always cost several times more than the second one’s. Why do the “small” processes appear in ‘nvidia-smi’

WechatIMG3.jpeg1116×342 77.1 KB


WechatIMG12.jpeg1120×190 49.7 KB