toch.cuda.amp.autocast() will use fp16 representation for allowed list of operations 3 regardless of the magnitude.
If your loss / gradient is too small, they will fall to 0.
Gradient scaling (torch.cuda.amp.GradScaler) is there to prevent gradients from being 0 if possible.
It

scales your loss with a large value (e.g.1024) and then
compute the gradients
unscale the gradient

If all the values required in the above process are in fp16 range, amp may work.
If any of them is larger than the fp16 maximum or smaller than the minimum, amp may not work.
You will have to find your optimal learning rate by running experiments and checking the gradient values.
You can refer to wikipedia page 4 to check the fp16 precision limitation.
I guess lr=10^-20 won’t work in fp16 for most of the cases.