Hello Yuri!



 yurib:

…
(btw, multiplication is done not even in fp32 but in full precision). So i’d guess (might be wrongly?) it’s just Pytorch which doesn’t support returning fp32 result


A quick comment on nVidia’s terminology:  I do not see anything
that makes clear what nVidia means by “full precision.”  I’d say
that this is either sloppy language or some subtle marketing
hype.  Because, bear in mind, that (other than possible exponent
underflow or overflow) multiplying two fp16’s as fp32’s gives
exactly the same result as multiplying them as fp64’s.  So (if
only for sanity’s sake) nVidia should have labelled their diagram
as “FP32 product”, rather than the potentially ambiguous “Full
precision product”.
(Also, it’s a little odd that their last step is “Convert to FP32
result”.  Last time I looked, the output of a “FP32 accumulator”
is a FP32 result.)
All that being said, it is true that you lose some information by
truncating your fp32 matrix multiplication back down to fp16.
It may be preferable not to.  However, the lesson of the numerical
analysts is that you get a lot of benefit (in certain realistically
common cases) from performing the multiply-accumulates in
fp32, and keep most of that benefit even after truncating back
down to fp16.  That is, performing the matrix multiplication in
fp16 gives you a fp16 result that has much less accuracy than
its fp16 precision might suggest, whereas multiplying in fp32
(and truncating back to fp16) can give you a result with near
full (or full) fp16 accuracy.
(The fp32 multiplication result may, in fact, be more accurate
than fp16 accuracy (or it could be less, in some cases), so
whether, in such a case, the loss of accuracy of truncating
back down to fp16 is outweighed by the speed up, is up to
you.  But broad experience teaches us that in many cases
it is.)
(I don’t know whether pytorch supports returning a fp32 result.
Your nVidia link suggests that the gpu gives you a choice.
If pytorch doesn’t give you a choice, it probably should, but
in many realistic cases it won’t be wasteful to return the result
as fp16.)
Best.
K. Frank