Hi! Thanks for answering!



 ptrblck:

This could point towards a high loss


That was my first idea, but, unfortunately, the loss value is average (~0.1 - 2), so there is no reason for the scaler to go that low in the first place. Also, it doesn’t stop decreasing, so that’s that.



 ptrblck:

it’s scaling value if NaN values are encountered.


The loss values are always valid. However, when I was running the script with torch.autograd.anomaly_detector, there was nans/infs in gradients every ~10th iteration, so that’s the reason for scaler to go down.