The blank predictions can indeed be produced by a highly imbalanced target distribution in the mask images and you could try to use a weighted loss function to counter this effect.
I’m unsure about the last effect of flipped predictions and then going to a blank prediction again.