Thanks for the example.
Are you using these RGB values directly for your classes?
If so, I would recommend to use a mapping such that each pixel contains only a class index in the range [0, nb_classes-1].