Hey, I have encountered a similar problem when I worked on an image colorization project. Some parts of the image were black and other parts seemed fine. I solved the problem by experimenting with different network architectures. The network on pix2pix paper worked fine if I remember correctly. Best of luck!