Are you using any data augmentation? If you do, turn off all augmentation schemes, and train exclusively on natural images. Goal is to make sure there is nothing wrong with the data augmentation script.
Also, visualize the images in your batch (input and output): to make sure that  the images that are fed to the model are consistent with what you expect.
I have experienced  similar “noisy” validation loss, and it originated either from a bug in the data augmentation, or in the batch generator.