@mcarilli with the upcoming RTX 3080/3090 introducing BF16 support (correct if I’m wrong), maybe I can skip gradient scaling and use your code as-is ?
However, how do I tell AMP that I want to use BF16 and not FP16 ?