Thanks for the info.
In that case I would stick to a classification criterion.
Could you post a sample mask with its values?
Usually your mask should have the shape [batch_size, h, w] and contain the class indices for each pixel.