Your model output of [4, 6] seems to be wrong for a segmentation use case, as it should be [batch_size, nb_classes, height, width].
Currently it seems your model outputs a single label for each input sample, so you might want to change the model architecture.