CUDA calls are generally asynchronous with respect to the host. They happen in-order on each GPU (within a stream). However, the As[i].cuda(device=i) and Bs[i] = Bs[i].cuda(device=i) will block the host.
There’s a limit to the number of outstanding calls, though (about 1022) after which they will block.
You can also stick the calls to gesv in separate threads.