How did the validation loss look during training? Did you see some overfitting observing the training and validation loss curves?
Also, does your training and validation set contain similar images with these specific colors?