I run Python script, and I see Python code, followed by C++ code.In C++ code, I find onEachDevice is called at the beginning of profiler start.
The phenomenon I saw is when I run Python code, I opened another terminal, and found four cards were running processes, but I didn’t set multi-cards and run multi-process.So I am confused.
Later, I put servel kernels on different kernels with ‘torch.cuda.set_device(0/1/2/3), y = y.to(“cuda:0/1/2/3”)’, and found that profiler running on serverl cards when tensors were put to different cards , and at last deal with the result by host thread.
At last, I want to summarize:profile with mulit-cuda cards will warm up all cards, and if kernels in python are put to different cards, the card will run a new process to run the kernel. Infomation such as tid will be collected in Event and return back to profile(Python) and show to user.
Is there anything wrong?
Thank you a lot!