Vibhatha_Abeykoon:

Here part A, and part B as shown in the comments must run concurrently to get the pipeline performance. Am I correct/wrong?


No. Because CUDA operations run asynchronously from CPU’s point of view, unless you explicitly call synchronize() on CPU. And they are inserted into the same CUDA stream on each device in this example, which will guarantee ops on the same device run in order, but ops on different device can run in parallel.



 Vibhatha_Abeykoon:

If so, as Pytorch eagerly executes the layers, it is not asynchronous? Am I following this right?


Yes, it launches the CUDA kernel right away, but the CUDA kernel execution can be asynchronous. And CUDA will actually queue the kernels in the stream and to coordinate the execution. So, “launch right away” does not mean it will wait for the kernel to finish, nor does it mean the the kernel will start on GPU immediately.



 Vibhatha_Abeykoon:

If both these clauses are true, having threads is required? Isn’t it?


No, with multiple CUDA device + async CUDA kernel behavior, you could still get parallel execution without threads. And actually, even with thread, all Python code on CPU will still run in sequentially due to GIL.