The segmentation mask in a multi-class classification use case should have the shape [batch_size, height, width] and contain the class indices in the range [0, nb_classes-1].
Does your target image contain 256 unique classes? If not, you should map each color value to a class index.
The output of your model should have the shape [batch_size, nb_classes, height, width].