Your model might be overfitting to the majority class.
Could you check the class distribution in your masks, i.e. how many pixels belong to which class?
If you have a lot of background (class0), your model might only lean this class and you could use e.g. focal loss to counter it.