The eps value is used here 4 to avoid dividing by a 0. value.
I just checked the state_dict of Adam using autocast and it seems all internal buffers are still stored in FP32, so I’m unsure why the eps value might cause trouble in this case (due to a potential underflow).
Here is a minimal code snippet:
model = nn.Linear(10, 10).cuda()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

with torch.cuda.amp.autocast():
    out = model(torch.randn(1, 10).cuda())

loss = out.mean()
loss.backward()

optimizer.step()
print(optimizer.state_dict())

@mcarilli are you familiar with similar issues using Adam?