Hi, sorry for reviving this thread after a year.
I have a similar question about adding a constant in the loss function: let’s suppose I want to multiply the final loss by a factor, and that factor is computed based on the inputs or the outputs within a batch.
In my case I’m working with triplet networks and a custom loss, and now I would like to add a scale factor based on the amount of ‘easy negatives’, for example. This is part of my loss function:
def forward(self, x, labels):
        ...

        l2_norm_positive = F.pairwise_distance(x[0], x[1],2)
        l2_norm_negative = F.pairwise_distance(x[0], x[2],2)
        easy_negatives = l2_norm_negative >= (l2_norm_positive + self.margin)
        easy_negatives_percent = torch.sum(easy_negatives).float()/labels[0].shape[0]
        scale_factor = 1/(1-easy_negatives_percent+self.eps)
        
        ...

If I remove the part where it computes the scale factor I get normal gradients after calling .backward(), but if I leave that section the gradients become nan.
How could I do that withouth breaking the computation graph? How should I deal with that constant (scale_factor)? Should I use .detach()?
Thanks in advance