I have a problem and I feel like the way I’m currently solving it is extremely inefficient so I’m hoping that it might be possible to speed this up.
The simplified set up is the following: I have some set of N parameters {x_i} which all go in to some complicated function. From the output of this function, I produce N different losses L_i (each of which can in principle depend on all of the initial parameters), but I want to optimise each parameter x_i with respect to its corresponding loss L_i only. What is the best way to do this?
Currently I am doing:
for i=1:N:
optimiser[i].zero_grad()
loss[i].backward(retain_graph=True)
optimiser[i].step()
but this is extremely slow. Is there a better way of doing this?