out1 = net1(in1)
out2 = net2(out1)

When I understand correctly, you said that operations are added to a stack and then given to the GPUs when computing resources become available. So, when you mentioned that the CUDA api is asynchronous by default, it would only run things in parallel if the inputs don’t have dependencies in the graph that were not previously satisfied?
Or in other words, in this code scenario above, even with 2 GPUs, it would wait until out1 is computed before out2 gets computed. That makes sense.
So, basically, the advantage of using 2 GPUs in PyTorch is that is simple extends the pool of computing resources available but does not affect the order in which the operations are processes. For instance, this could be useful if e.g., one giant matrix multiplication would be too large for one GPU, so the computing resources from the 2 GPUs could be pooled to compute the dot products in parallel?