Yes, that’s pretty much what I’m doing. Recently I have changed the image patch size to 256 instead of 96. I did this because of PyTorch’s pretrained VGG19, as it asks for a input of size higher than 224, but it did not improve much my results, so I guess the problem is in my code. I’ll be reviewing the code as a whole tomorrow, in hope of finding a mistake and fixing it.
The good part is that I can reach the reported metrics of SRResNet-MSE, which makes me suspect that the problem is related to my VGG Loss implementation.