i first need to move  a1  to cuda0 or  a0  to cuda 1. Which throws cuda out of memory. However, the result of  a0 += a1  if computed can hold in cuda0 or cuda1.

If a0 and a1 don’t both fit on a single GPU, you could try putting part of each tensor each GPU, add each chunk, and then concatenate the results with torch.cat. i.e. split each tensor in half, put the first half of each tensor on ‘cuda:0’, add them, then concatenate the result of doing the same with the second half of the tensor.