Hi,
My test code is like this:
a = torch.randn(16, 3, 16, 16)
a1 = a.cuda(1)
a2 = a.cuda(2)
a3 = a.cuda(3)
a4 = a.cuda(4)
torch.cuda.synchronize()

a = torch.randn(16, 3, 1204, 1024)
t1 = time.time()
a1 = a.cuda(1)
a2 = a.cuda(2)
a3 = a.cuda(3)
a4 = a.cuda(4)
t2 = time.time()
torch.cuda.synchronize()
print(t2 - t1)

a = torch.randn(16, 3, 1204, 1024)
t3 = time.time()
a1 = a.cuda(1)
t4 = time.time()
torch.cuda.synchronize()
print(t4 - t3)

The results is:
0.18216228485107422
0.03782033920288086

Which means that copying one cpu tensor to one gpu tensor is faster than copying same tensor to multi-gpu, the process seems to be a one-by-one operation. How could I make the copy in parallel please?