Normalization layers, such as batch normalization and group normalization, comprise of reduce operations which, as Nvidia’s Apex suggests 11, should be done in full FP32.
However, I have not seen these layers’ operations in the Ops eligibility 6 page. Hence, the behavior of these layers are non-changing which would result in FP16 outputs.
Is this a valid concern? It was not mentioned in the documentation.