@ptrblck thanks for your reply!  I used the DataParallel module, and after I checked the code, I found that the GPU0 will gather the results from other GPUs like GPU1,2 and then it processes some computations by itself and then scatter to others. So I want to check the network between them (the way of communication). And how the GPU communicate with the CPU in this DataParallel approach. I tried to dive into the C++ code, however, I am not clear where could I search for this.
Thanks