If I do something like this:
# GPU 0:
loss_c = a * b 

# GPU 1:
loss_f = d * e 

And then I add them together (converting one output to the other’s GPU):
total_loss = loss_c + loss_f.cuda(0)

Then when I run backward() on total_loss, will the backward pass split off onto both GPUs? Or just will it just take place on GPU 0?