Thank you., I managed to make it work up to some point. Now, the data in cuda:0 need to be added to the data in cuda:1. If i moved the data in cuda0 to cuda1 before the addition, it throw CUDA out of memory.
Let say, if i have
a0 = torch.randn(4, 4) # in cuda 0
a1 = torch.randn(4, 4) # in cuda 1

To compute a0 += a1, i first need to move a1 to cuda0 or a0 to cuda 1. Which throws cuda out of memory. However, the result of a0 += a1 if computed can hold in cuda0 or cuda1.
Is there a way to solve this problem. I’m using our University GPU , a Tesla K80
Thank you.