Hi,
I have a loss that is computed on 2 GPUs and is stored in list called ll_list. At the end I gather them and add them on the device 0 and I run my backward.
However it seems to me that there are two ways to do that.
The first one consists of doing :

ll = sum([ll.to(‘cuda:0’) for ll in ll_list])

Which I’m currently doing and works fine.
However I noticed that it is way faster to do :

ll = torch.cuda.comm.reduce_add(ll_list)

But then my model doesn’t converge anymore and the loss is going crazy.
Does someone has an explanation about that ?
Thanks