It might depend how the model performance and loss are calculated. E.g. if your model outputs the same class for all pixels during validation without significantly changing its confidence, the loss and performance might just get stuck.