Colorizing black and white photos with deep learning
The 'averaging' problem is interesting and I think related to the Euclidean-style loss function: it is correctly minimizing the color distance between its gray-brown guess and all the possible shades? If that's what's going on, then messing with the architecture won't fix it because it's already reaching the 'right' answer and what is necessary is a better loss... Perhaps something like DCGAN where the loss function is another CNN which is being trained to guess whether an image is original or generated? (So you'd feed the BW image into the generator, which converts it into a colorized image; the colorized and the true color images are both fed into the discriminator, which tries to guess which is the real one. Then error is backpropagated into both.)