Is there any minimum learning rate, for training with amp (mixed precision training). Say if lr scheduler drops the lr to a high precision value (say 10^-20) then will the weight get updated in the that epoch?
I understand that loss and some other operators are calculated in fp32 during mixed-precision training, so they may not have much impact, but how the operators that use fp16, will the weight update stop for them?