I’m not sure if your NAS implementation initializes the model parameters with “large” values explicitly or if the general training of these new architectures tends to create large output values, which might cause overflows easily. In that case I’m unsure if there is a better workaround than to increase the eps value, as 1e-8 might underflow in FP16 if you add it to a larger tensors.