Well my first question was during the backwards pass, I want the largest distance to have the highest weighted gradient. However, if you multiply that by a constant, won’t it be divided out when calculating the gradient? As for if they’re differentiable or not, it seems to be working as intended, and I’ve tried to verify that the graph is unbroken, so I don’t think its necessarily an issue.
Here is my code (it also has a permutation part since I’m trying to find order invariant minimums along several axes)
    # x = [n x 3 x 3]
    # y = [n x 3 x 3]
    x,y = x.view(-1,3,3),y.view(-1,3,3)
    xx = torch.stack([x]*6,1)
    for i,p in enumerate(itertools.permutations([0,1,2], 3)):
        for j,c in enumerate(p):
            xx[:,i,j] = x[:,c]
    xx = xx.unsqueeze(1).expand(-1,xx.size(0),-1,-1,-1)
    # xx = [n x n x 6 x 3 x 3]
    yy = y.unsqueeze(1).expand(-1,6,-1,-1)
    zz = yy-xx
    dnorms = torch.norm(zz,2,-1)
    dmins = torch.min(dnorms,-1)[0]
    dmaxs = torch.max(dnorms,-1)[0]
    #heres the weighting part
    dd = torch.sum(dnorms,-1)*2 + dmaxs - dmins
    # dd[i,j,k] = distance from x[i] to y[j] for permutation k
    # find minimum over all permutations
    pd,_ = torch.min(dd,-1)
    # minimum distance from x to y
    xmin,_ = torch.min(pd,0)
    # minimum distance from y to x
    ymin,_ = torch.min(pd,1)
    return torch.mean(xmin) + torch.mean(ymin)```