mrartemev:

During the training, the scaler’s scale is decreasing from its usual values to very low ones (like 1e-7).
Strangely, it happens only for critic and only for WGAN-GP model. When I try LS-GAN or a simple GAN everything is ok.


This could point towards a high loss and isn’t a problem by itself, if the loss scaler stops decreasing the value after a while and thereafter does it sporadically.



 mrartemev:

So, the problem is somehow inside the GP part, but I just can’t locate it. (for some reason, fp16 grads for GP contains nans, but I have no idea why)


This is expected, if the loss scaling is too high and the loss scaler will reduce it’s scaling value if NaN values are encountered.
However, the loss value should never become invalid (NaN/Inf). Is this the case for your model?