Hi,
I’m trying to implement a custom regularizer, but my final loss diverges (like exp), my custom loss (and regularizer) function is :
jacobian_similarity = [[1, -1, 0], [1, 0, -1], [0, 1, -1]] #for 3 classes
def customized_loss(pred, target, x):
    def mean_square_loss(pred, target):
        return (((pred - target) ** 2).sum()) / pred.data.nelement()
        
    def regularizer(pred, x):
        r = 0
        for x_ in range(batch_size):
            for js in jacobian_similarity:
                pred[x_].backward(torch.FloatTensor([js]), create_graph=True)
                r += torch.norm(x.grad[x_], 2)
                x.grad.data.zero_()
        return r
    
    loss = mean_square_loss(pred, target)
    regu = regularizer(pred, x)

    return loss + 0.01*regu

jacobian_similarity is a list who has all possible ‘differences’, so when .backward() is calling with one of it inside it will compute the difference of jacobians rows, e.g.
with [1, -1, 0] i will have, jacobian_f1 - jacobian_f2.
My regularizer steps are :

compute jacobian of each output wrt the input
for each row in jacobian (f1,…,fi, … ,fn) compute the norm2 with all other rows (like norm2(f1,f2) + norm2(f1,f3) + … + norm2(f1,fn) + norm2(f2,f1) + …)
add regularizer to the loss

Maybe my implementation of regularizer disconnect the graph somewhere ?