Let’s say I have 8 models hosted on 8 GPUs (same class, different initialization)
models = [MyModule().cuda(i) for i in range(8)]

And I have a CPU tensor
x = torch.randn(1000, 128)

If I run the forward pass for all 8 models in a for loop like this
predictions = [models[i](x.cuda(i, non_blocking=True)) for i in range(8)]

The run time is significantly slower, probably 6x~7x than just running it on one single GPU
models[0](x.cuda(0, non_blocking=True))

Is this expected? I was under the impression that CUDA operations are asynchronous, and it should remain non-blocking unless the result of the forward pass is used by someone else?