Is there more to this timing script or is this the bulk of it? One issue that could be happening here is that the first run of the linear layer pulls in cuBLAS (and might do some light autotuning as well) on each GPU. It could be that these steps are synchronous and execute one at a time, incurring the serial time penalty you see here. Does the same thing happen when you run the linear layer on each GPU a few times as a “warmup” before doing the timing>
Note that you would want to do torch.cuda.synchronize before starting the timing and also before stopping the timing.