Zhang_Chi:

so out2 = F.conv2d(in2, weight2) will start before F.conv2d(in1, weight1) is finished on GPU 1, right?


If the CPU is fast enough to schedule it and the actual kernel execution takes more time than the launch, then yes.
You won’t be able to see any overlap with a tiny workload, as the kernel launch overheads would be larger than the actual GPU workload, i.e. kernel1 finishes before the CPU can schedule the launch of kernel2.



 Zhang_Chi:

BTW, what if the weights are shared in two lines, should I do it by:


Yes, you have to move the parameters to the appropriate device before executing the operation.