yes, in general, each GPU wont wait for the previous GPU at all, unless allocations are occuring (like malloc / free). The CUDA API is asynchronous, so snippets such as:
C_ = []
for i in range(ngpu):
    C_.append(torch.matmul(A[i], B_[i]))

happen in parallel.
There are some subtle cases where you have to switch the CUDA stream, but I dont think you’ll run into these cases unless you are doing something very very specific.