mrartemev:

However, when I was running the script with torch.autograd.anomaly_detector, there was nans/infs in gradients every ~10th iteration, so that’s the reason for scaler to go down.


That might be expected and doesn’t necessarily point towards a broken training.
A high loss could create large gradients, which might overflow, but if that’s not the case also high parameters might create these large gradients as seen here:
# high loss
model = nn.Sequential(
    nn.Linear(1, 1, bias=False),
    nn.Linear(1, 1, bias=False))
data = torch.randn(1, 1)
out = model(data)
loss = out.mean() * 1000.
loss.backward()
print(loss.item())
print(model[0].weight.grad)
print(model[1].weight.grad)


# large parameter values
model = nn.Sequential(
    nn.Linear(1, 1, bias=False),
    nn.Linear(1, 1, bias=False))
with torch.no_grad():
    model[1].weight *= 1000.
data = torch.randn(1, 1)
out = model(data)
loss = out.mean()
loss.backward()
print(loss.item())
print(model[0].weight.grad)
print(model[1].weight.grad)

Are you only concerned about the loss scaler value or is your training not converging in any way?