I guess your mask has some kind of color code in RGB, e.g. red ([255, 0, 0]) means car, while blue ([0, 0, 255]) means building.
If that’s the case, could you post a sample image or post the mapping directly, since we would need to transform the RGB color-coded mask into a class index mask.