I’m working on a neural network to predict an aperture which is rotating. The output values are the machine parameters of this aperture, but for calculating the loss, I translate these to an image in order to calculate a effective targeted 3D volume. Unfortunately, my algorithm for creating these images is not differentiable and I do not obtain any gradients for these. Depending on the output, image pixels are open (one) or closed (zero) (last line of code basically).
I’m using pytorch-lightning, but I do not think this is related to my problem since this index operation is not differentiable.
    img = torch.zeros(IMG_SIZE)
    
    # isocenter as image origin
    isocenter = 0.5 * torch.Tensor(IMG_SIZE)
        
    # calc image positions for every leaf pair. open 1 closed 0
    for leaf_index in range(NUM_LEAFS):
        leaf_pair = leaf[leaf_index, :]
        leaf_pair = torch.round(leaf_pair)
        x1 = isocenter[1] + leaf_pair[0] * IMG_RES[1]
        x2 = isocenter[1] + leaf_pair[1] * IMG_RES[1]
        
        x1 = x1.clamp(0, IMG_SIZE[1] - 1).int()
        x2 = x2.clamp(0, IMG_SIZE[1] - 1).int()
        y1 = leaf_index * IMG_RES[0]
        y2 = y1 + IMG_RES[0]
        img[y1:y2, x1:x2] = 1 
    

Does anyone have an idea how to overcome this problem? Any help is much appreciated, thanks!