Sorry for the really long delay in my response. I got carried away in some other projects. However, here is my customized loss function for the training pipeline:
def dice_loss(pred, target, threshold = 0.5, smooth = 1e-6):
    pred = torch.sigmoid(pred)
    pred = (pred>threshold).float()
    num = pred.size(0)
    m1 = pred.view(num, -1)  # Flatten
    m2 = target.view(num, -1)  # Flatten
    intersection = (m1 * m2).sum()
    dice_score = (2. * intersection + smooth) / (m1.sum() + m2.sum() + smooth)
    return 1 - dice_score

I think I’m hurting the gradients by applying the threshold. Let me know if am right or not. I’ve removed the threshold part and now the model seems to converge.