FYI, I ended up asking the Lightning guys about the scaler causing overflow and they wrote:

At the beginning of training, in the first few iterations it can happen that the grads become inf or nan. AMP will skip the optimizer.step() in that case and adjust the scale factor iteratively until it converges to a point where that doesn’t happen.

See: AMP scaler always causes backwards pass to overflow · Issue #9799 · PyTorchLightning/pytorch-lightning · GitHub 7
I hope this helps other people who run across this in the future. So apparently you can’t enable torch.autograd.set_detect_anomaly(True) on first epoch…