I discovered this also myself and am already using an additional classifier. Although this does indeed increase the performance, it may fail for cases where Class A and B are present on the image and both get labeled to be Class A.
By the way, it seems like that the additional classification loss alone improves the segmentation results even when the classification result is not used for calculating the final output. I think it just helps to learn better filters in the encoder.