Thanks for the info.
Your code seems to work.
def my_loss_fn(output, target):
    N = output.size(0)
    for i in range (N): 
        
        Y = output.clone()[i,0]
        X = target.clone()[i,0]
        X_comp = 1-X
        Z_num1 = torch.max(X,X_comp)
        Z_den = 2*torch.max(X,X_comp)
        Z_1 = Y/Z_den
        Z_2 = (Z_num1-X)/Z_den
        Z = Z_1+Z_2
        Z_comp = 1-Z
        
        F = -X*(Z*torch.log(Z)+Z_comp*torch.log(Z_comp))
    loss = (1-(torch.sum(F)/N))
     
    return loss


model = nn.Sequential(
    nn.Linear(20, 10),
    nn.ReLU(),
    nn.Linear(10, 2),
    nn.Softmax(dim=1)
)

x = torch.randn(1, 20)
target = torch.randn(1, 20)
output = model(x)
loss = my_loss_fn(output, target)
loss.backward()
print(model[0].weight.grad)
> tensor([[ 0.0002,  0.0013,  0.0000, -0.0011,  0.0003, -0.0004,  0.0013,  0.0010,
         -0.0010, -0.0010, -0.0004, -0.0016, -0.0008,  0.0005,  0.0008,  0.0021,
          0.0006,  0.0019,  0.0005, -0.0015],...

I’m not sure, why you are iterating the batch dim and just use the last F.
Also, I don’t think you need to clone the output and target.
Does this code work for you?