I think the more fundamental problem is that the program is trying to minimize error where error is defined as deviation in color from the original image. This means objects that can be many different colors but are always strong and saturated average to a brown, as you said. And as you said, for those types of objects, it&#x27;s best to pick a random color and make it bright and saturated. The best way to have it do that is to redefine error as some metric of how &quot;realistic&quot; the picture looks vs the original. For example, a picture of a car recolored to look bright blue looks similarly &quot;realistic&quot; to the human eye as the original picture where the car is bright red. The deviation in color is high, but it still looks &quot;good&quot;, so the error should be low. I have no idea how this metric would be calculated without humans evaluating the output manually, though.