I use a pretrained vgg16, extract the feature maps before the last three pooling layer,and passed through a inception module to fuse the three feature maps to one, them send this output as decoder input, upsampling 3 times to recover to initial size.
U mean use only one pic to train? I used to use 30 pics to train 300 epoches ,but just generate a black-white pic like a heatmap, with no obvious shape, just like a scattering plot.and the loss seems convere at 0.3 when 30/300,  what a early converge! then it fluctuates, no more obvious change.