Hi all,
I am training a GAN for time series prediction. During training, I sample n_samples predictions and backpropagate the L2-loss for the best sample.
for _ in range(n_samples):
    out = net(x)    
    loss = lossfunc(x,y)
    l2_loss = l2_loss.append(loss)

l2_loss.stack( l2_loss ) 
min_loss = torch.min( l2_loss )
min_loss.backward()

When I call min_loss.backward(), the time for calculating the backward pass grows linear with n_samples. However, the gradient for the non minimum losses should be 0 and not considered for optimisation.  I only want to backpropagate the L2-loss for the best sample.
Therefore, I do not understand why the calculation time increases and does not stay constant? Does .backward() also calculate all gradients even if they are not used? Is there a way to make my code more efficient and detach all non minimum losses?
Thank you for your answer:)