Pinterest, Engineering, Monetization.
models use the pytorch default float32 datatype, but tensor cores don’t operate on this. To get an inference speedup, we need to use a lower-precision data type, of which pytorch offers two easy options: float16 and bfloat16. Both of these data types use 16 bits instead of 32 to represent a number,