I found this thread interesting. On a related note:
cuda(non_blocking=True)

has a quite significant delay (~50% of a blocking call)
Is there anyway to reduce the delay of the non-blocking call?
This is critical to the data transfer from CPU to multiple GPUs. Ideally we’d like to do data transfers in parallel, e.g.
x = torch.randn(1000, 1000, 1000)
xs = [x.cuda(i, non_blocking=True) for i in range(8)]