So here’s what I am trying to do:
I have an original model_0 and N copies of it model_{1,…,N} that are trained on different partitions of the data.
I want to compute losses for model 1 through N, then use their summed losses as the loss for model_0.
What I do now is compute the sum of losses in a variable loss_total
then I make a forward pass through model_0, and manually change the data variable to the wanted loss value
total_losses = #sum of losses from N copies
optimizer_0 = torch.optim.Adam(model_0.parameters(), lr=lr)

model_loss = loss(model_0(x_random), y_random) # make a forward pass through any data

model_loss.data = total_losses # manually change the loss

optimizer_0.zero_grad()
model_loss.backward()
optimizer_0.step()

I am not  sure that this is the way to go about it, so I appreciate any expert opinion on the matter.
Thanks,