I am working on a GAN for super-resolution at the moment (SRGAN 1). This GAN’s generator is initialized as a previously trained CNN. In summary, one should train the generator first, and then train the GAN with a decent generator initialization. I have trained the CNN and its results are as expected. Now, when training the GAN, I am not getting the expected results and the generated images introduced some artifacts like this:

High-resolution image

Super-resolution image (CNN)

Super-resolution image (GAN)
The training setup is, in summary:

Train the generator (optimized for MSELoss);
Train the GAN (optimized for Perceptual Loss, as described in the paper), initializing the generator with the results from the first step. The optimizer here is the same from 1. (Adam, LR=0.001).

Among other things, I am thinking it might be a problem with the generator optimizer. After training the CNN separately, I didn’t save the optimizer’s state_dict and started training the GAN with a new optimizer, but with the same parameters. I thought this was right because the generator should be optimized with a different loss function, but I’m not sure. Does someone have an input on why this problem is happening? Do you think it can be related to the optimizer or anything else?