The &#x27;averaging&#x27; problem is interesting and I think related to the Euclidean-style loss function: it is correctly minimizing the color distance between its gray-brown guess and all the possible shades? If that&#x27;s what&#x27;s going on, then messing with the architecture won&#x27;t fix it because it&#x27;s already reaching the &#x27;right&#x27; answer and what is necessary is a better loss... Perhaps something like DCGAN where the loss function is another CNN which is being trained to guess whether an image is original or generated? (So you&#x27;d feed the BW image into the generator, which converts it into a colorized image; the colorized and the true color images are both fed into the discriminator, which tries to guess which is the real one. Then error is backpropagated into both.)