I was using Content Loss (features) and Adversarial Loss to train the GAN. In the paper, the author calculates the Content Loss either by comparing the generated image (SR image) with the original image (HR image), using MSE Loss, or by comparing the feature maps generated when feeding the low-resolution image (LR image) and HR image through the VGG, also using MSE Loss.
On the other hand, it is not clear if the Content Loss is exclusive. I mean, one may ask if Content Loss for the GAN is calculated only using the VGG feature maps (Case 1) or if it uses the VGG features maps AND the MSELoss(SRimage, HRimage) (Case 2).
I have tested briefly using both for the Content Loss. Training with the described Case 1 generates the bad images I originally mentioned (those with the artifacts and low PSNR). The Case 2 on the other hand looks promising, as it holds to an PSNR level similar to the reported in the paper, but I didn’t finish training because it seems to me this was not originally done by the author, because of the following statement in the paper:

Instead of relying on pixel-wise losses we build on the ideas of Gatys et al. [19], Bruna et al. [5] and Johnson et al. [33] and use a loss function that is closer to perceptual similarity. We define the VGG loss based on the ReLU activation layers of the pre-trained 19 layer VGG network described in Simonyan and Zisserman [49].

This statement makes me think that Case 1 is the right one, even though I’m not reaching the expected results just yet.
Also, maybe I’m being too strict and the instead just means that it will not rely only on pixel-wise losses, but pixel-wise and VGG loss, which would make the Case 2 the right one. Let me know what you think, any input is good, this is a very important work for me.