While I have reservations about PyTorch enabling tf32 by default, note that this



 KFrank:

tf32 is essentially half-precision floating-point


is a bit of an oversimplification, I’d probably describe it as “tf32 has the dynamic range of fp32 but the relative precision of fp16”. For many applications, dynamic range of fp16 has been troublesome (so half overflowing to +/-inf or underflowing to 0), this is why AMP does the gradient scaling for backpropagation. In place where fp16 fails due to the lack of relative precision, tf32 won’t cut the mustard, either.
If tf32s characteristics are acceptable, it does seem to offer a formidable speed-up.
Best regards
Thomas