The loss was decreasing even before, and the prediction was getting worse!

    print("prediction:")
    unique, counts = np.unique(prediction[0].byte().cpu().numpy(), return_counts=True)
    print(dict(zip(unique, counts)))
    unique, counts = np.unique(prediction[1].byte().cpu().numpy(), return_counts=True)
    print(dict(zip(unique, counts)))


I’m printing the prediction to see what’s going on in each step, and after a point all the elements in each class(prediction[0] or prediction[1]) turn zero.