In the actual classifier I’m using, I apply a Sigmoid() in the last layer, so that the classifier outputs are between 0 and 1. The custom loss function I’m trying to implement is supposed to be an extra regularization loss function. For the main loss function I use BCEWithLogitsLoss(). For the extra custom regularization loss function I then wanted to take the predicted values and put them in the same format as the targets, so putting all predicted[predicted < 0.5] = 0 and all predicted[predicted >= 0.5] = 1. I tested this again, when I don’t apply these changes into my custom loss function I also get gradients, like you did during your iterations. However, when I add this change, I just get 0 gradients again, no matter how often I run the program. Shouldn’t I at least still get gradients through the values that were changed to 1? I’m a bit confused about this.
def custom_loss(preds, t):
    total = torch.zeros(1, dtype=torch.float, requires_grad=True)

    p = preds.clone()
    p[preds < 0.5] = 0
    p[preds >= 0.5] = 1

    for pair_p2, pair_t2 in zip(torch.combinations(p, r=2), torch.combinations(t, r=2)):
        pair_p, _ = torch.sort(pair_p2)
        pair_t, _ = torch.sort(pair_t2)
        pair_p = pair_p.double()
        pair_t = pair_t.double()
        
        g1 = (pair_p[0] == pair_p[1]).type(torch.uint8)
        g2 = (pair_t[0] == pair_t[1]).type(torch.uint8)
        total = total + (torch.max(pair_t[g2.item()], g2.type(torch.double)) - torch.max(pair_p[g1.item()], g1.type(torch.double)))**2
    return total