jbohnslav:

However, this seems to make little sense to me. The relevant portions of my loss function have been carefully scaled by prior work and my own work, so scaling them individually seems like it would ruin that. Furthermore, having a backwards pass for each component of my loss function would take much longer.


The GradScaler will scale the loss to increase the magnitude of the gradients and will unscale it using the same scaling factor. Your manual scaling won’t be changed, so I would recommend to try it out and check, if it solves the issue.