Thanks for the answer, I think my custom loss operation is not differentiable, but wouldn’t that throw errors? Currently, my code runs without any errors.
for index, item in enumerate(output):      
            temp = item.max(dim=1)[1]
            temp = temp.cpu().numpy().squeeze()
            prev = np.nan
            change = []
            for index2, item2 in enumerate(temp):
                if index2 == 0:
                    prev = item2
                if prev == 1 and item2 == 0:
                    change.append(index2)
                prev = item2
                
            change_np = np.array(change)
            label2_np = np.array(label2[index])
            if len(change)>len(label2[index]):
                #padd label
                tt = len(change)-len(label2[index])
                label2_np = np.concatenate((label2_np, np.zeros(tt)))
            else:
                #padd output
                tt2 = len(label2[index])-len(change)
                change_np = np.concatenate((change_np, np.zeros(tt2)))
            
            loss_temp = sm_l1_loss(torch.Tensor(change_np), torch.Tensor(label2_np))
            loss_arr.append(loss_temp)

Here is what I did. Thank you so much again!