You are right that the data transfer is indeed not 100% parallel. But even after I remove the data transfer from the comparison. The computation on multiple GPUs is still significantly (~6x) longer
model = torch.nn.Sequential(*[torch.nn.Linear(1000, 1000) for i in range(10)]).cuda(0)
x = torch.randn(100000, 1000).cuda(0)
timeit -r1 -n1 model(x)
1 loop, best of 1: 820 µs per loop

In [32]: models = [torch.nn.Sequential(*[torch.nn.Linear(1000, 1000) for i in range(10)]).cuda(i) for i in range(8)]
In [33]: xs = [x.cuda(i) for i in range(8)] # pre-distribute x to 8 GPUs
In [34]: timeit -r1 -n1 [model(x) for model, x in zip(models, xs)] # Is this supposed to run in parallel?
1 loop, best of 1: 5.17 ms per loop. # ~6x longer