Solved by nairbv in post #7 
              
              
                i first need to move  a1  to cuda0 or  a0  to cuda 1. Which throws cuda out of memory. However, the result of  a0 += a1  if computed can hold in cuda0 or cuda1. 

If a0 and a1 don’t both fit on a single GPU, you could try putting part of each tensor each GPU, add each chunk, and then concatenate th…