I don’t currently have the AMP code in the repo (because of this bug), but I’m using the ESRGAN model for 4x SISR (RRDBNet: https://github.com/victorca25/BasicSR/blob/3fb5851af3a411df153f0b0e5873f3b2de324a3d/codes/models/modules/architectures/RRDBNet_arch.py 1), the inputs are images of size 32x32 and the target patch size is 128x128. The loss function is the contextual loss (https://github.com/victorca25/BasicSR/blob/3fb5851af3a411df153f0b0e5873f3b2de324a3d/codes/models/modules/loss.py#L767 1). I’m not using other losses or a discriminator, only a generator. The model “trainer” is here: https://github.com/victorca25/BasicSR/blob/3fb5851af3a411df153f0b0e5873f3b2de324a3d/codes/models/SRRaGAN_model.py 1 (minus the AMP parts), but its really no different from the official documentation.
I know it may not be easy to reproduce like this, but maybe I could help with some guidance? I already tried changing the default GradScaler() parameter initialization, increasing the backoff_factor only delays the inevitable and the others can’t really change anything.