Hi Yuri!



 yurib:

…
Is it possible to multiply two fp16 tensors but get output in fp32?
…
According to nvidia accumulation is done in fp32 so seems wasteful (in terms of performance) to return just fp16 as an output?


It’s well known in the numerical analysis community that a
significant increase in accuracy can be achieve for vector
dot-products (i.e., for matrix multiplication) when extended
precision (that is, for your case, fp32 is “extended precision”
relative to fp16) is used for the “accumuland” of the dot-product
multiply-accumulate chain.  (This is especially true for large
matrices.)
(I don’t have a good reference for this off hand, but if you
search for the rationale behind the 80-bit internal precision
of the Intel 8087 math coprocessor, you should be able to
find lots of good stuff.)
So nVidia is making a potentially sensible trade-off here.  You
(potentially) get the most bang for the buck by performing the
tensor multiplication (multiply-accumulate) in extended precision
(fp32), even if you only return a regular precision (fp16) result.
Best regards.
K. Frank